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Executive Summary 


Purpose 

A human-in-the-loop (HITL) simulation was conducted in May 2010 to determine the feasibility and value of 
conducting multi-sector planning (MSP) operations in a mixed equipage environment. Aircraft were categorized as 
equipped or unequipped based on the presence or absence of an air-ground data communications (Data Comm) 
capability for receiving auto-loadable clearances and transfer of communication messages from the air navigation 
service provider (ANSP). The purpose of the study was to determine the feasibility and possible benefits of 
introducing multi-sector planning in a mixed equipage context, or whether Data Comm equipage was required for 
MSP operations. Each test scenario presented one of three different equipage levels to the controllers (10%, 50% 
or 90% equipped aircraft), so that the operational impact of different equipage levels could be observed. 

Operational feasibility assessment addressed two related questions: (1) are MSP operations feasible for 
unequipped aircraft, and (2) are they feasible in a mixed equipage context. Similarly, two categories of potential 
benefits were explored: (1) system performance improvements (e.g., throughput, workload) associated with MSP 
at different equipage levels, and (2) the possibility of providing differential service for equipage through MSP 
operations. Tool requirements (for both planning and controller stations), as well as planning and coordination 
procedures - within facility (traffic management unit/operational area) and within sector (R-Side/D-Side) - were 
two other topics addressed in the study. 

Findings 

The following list summarizes the experimental findings covered in this report, along with the results that support 
them. 

1. MSP operations were feasible in a mixed equipage environment. 

o Trajectory coordination was demonstrated to be feasible for unequipped aircraft. Planners (traffic 
management coordinators or area supervisors) and controllers sent a total of 1595 coordinated 
clearance requests during the simulation, 1026 for unequipped aircraft. Out of the 1595 requests, 
1574 were accepted and executed by their receiver (3.3.1). 

o Mixed equipage operations were feasible. Traffic management coordinators (TMCs) did a good job of 
managing controller task load, and were able to effectively balance throughput with sector 
complexity /task load at each equipage level (3.2.1, Figure 14; 3.5.1, Figure 26). 

o Operator feedback indicated that operations for mixed equipage multi-sector planning, and for 
unequipped aircraft trajectory coordinated clearances, were feasible and acceptable at all equipage 
levels. In general, post-run ratings of different task load measures were increasingly positive with 
increasing equipage levels (3.5.3, Figure 27, Table 12). 

Some caveats: 

o When explicitly asked in post-run questionnaires, some operators reported equipage-related 
confusion during the 50% runs. The 50% condition also received the highest mean post-run 
"frustration" rating from all participants, and the worst ratings from area supervisors on most of the 
other post-run task load measures (Table 12, 3.5.2- 3.5.4). 
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o There were 7 cases where the wrong input action was used to execute a clearance for an equipped 
aircraft. Although input errors were also observed on other tasks, mixed equipage operations may 
have been a contributing factor to these particular errors, since the controllers had primarily been 
working on unequipped aircraft when they made these mistakes. (3.3.5, Tables 8 and 9). 

2. Multi-sector planning operations supported priority handling of equipped aircraft. 

o Equipped aircraft were allowed greater access to congested airspace, showing consistently higher 
test sector throughput for equipped aircraft was observed when compared to unequipped aircraft, 
indicating that planners were selectively rerouting unequipped flights to reduce test sector 
complexity. (Results section 3.2.1, Figure 15 and Table 3). 

o Controllers moved unequipped aircraft more frequently when solving a mixed equipage conflict, 
allowing the equipped aircraft to fly its original route without interruption. (3.3.3, Figure 21). 

o Flight path increases were larger for unequipped aircraft in all runs at all equipage levels (3.2.2, Figure 
19). 

3. Higher equipage levels resulted in higher sector throughput with lower controller workload. 

o Total test airspace throughput increased with increasing equipage levels. (3.2.1, Figure 14). 

o Although, on average, radar controller and radar associate workload was acceptable across all three 
equipage levels, their workload decreased as equipage level increased. (3.5.1, Figure 26). 

o TMC and area supervisor workload was lowest at the highest equipage level (90%), and on average 
was acceptable across equipage levels (3.5.1, Figure 26). 

4. Tools and procedures were effective and satisfactory for mixed equipage operations. 

o Both TMCs and controllers developed the majority of their coordinated clearances (CCs) for 
unequipped aircraft. 219 of the 229 CCs sent to test sectors were accepted (3.3.1). 

o Most CCs received by test sectors were sent from other controller positions (166 of 229). Controllers 
used the CC function for both within sector and between sector clearance coordination (3.3.3). 

o Participant self-ratings of workload and performance were satisfactory at all equipage levels (3.5.2, 
Table 12). 

o TMCs gave usability and usefulness ratings of 5.5 to 6, on a 1 (not usable/useful) to 6 (very 

usable/useful) scale, for coordinated clearance and trial planning functions for both equipage types 
(3.6.2, Figure 31, Figure 32, & Figure 33). 

o Controllers gave positive usefulness and usability ratings to all of the key tools for developing and 
coordinating clearances for both equipped and unequipped aircraft (3.6.1, Figure 28, Figure 29). 

5. Insights were gained on both R-D and TMU-Supervisor task distribution. 

o TMCs performed far more multi-sector trial planning and coordinated clearance (CC) functions 
(sending more than 13 times as many CC requests) than area supervisors (3.3.1., Table 5). 
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o The supervisory TMC determined, divided, and assigned MSP responsibilities between the other two 
TMCs and took care of within and between Center verbal coordination (3.5.6). 

o TMCs' multi-sector planning roles were most often divided by geographic area, task (e.g., weather vs. 
volume), or altitude strata. TMCs reported that division by altitude strata seemed to result in the 
least duplication of effort (3.5.6). 

o The radar associate routinely used the tools to trial plan and send clearances for unequipped aircraft 
to his own radar controller to voice up to the aircraft (3.3.4, Tables 6 and 7, 3.5.6, Tables 13 and 14). 

o Division of other duties between radar controller - radar associate pairs was more varied, and 
depended on radar controller personal preferences, and with equipage level (3.5.6). 

Major Conclusions 

Overall, the results suggested that MSP operations were feasible in a mixed equipage environment and that the 
MSP tools were effective with both equipped and unequipped aircraft. Using the MSP tools, traffic management 
coordinators were able to manage controller task load, effectively balancing throughput with complexity and 
controller task load at each of the three equipage levels tested. Also across equipage levels, mean reported task 
and workload remained tolerable and operational acceptability was reported to be satisfactory. Although reported 
frustration and confusion were comparatively higher at the 50% equipage level than at the other levels, overall the 
50% mix was believed to be workable. 

Benefits were observed both in terms of system performance and operational support for a best equipped best 
served (BEBS) policy of air traffic management. As equipage level increased, throughput increased, even as 
controller workload decreased. MSP operations effectively supported priority service for equipped aircraft; more 
equipped than unequipped aircraft were routed through the test airspace and unequipped aircraft received a 
greater increase in flight path length. 

Other operational procedures established throughout the simulation suggested that the bulk of multi-sector 
planning - that is, trial planning and clearance coordination - can be effectively carried out by the traffic 
management unit (TMU), with operational area supervisors performing these functions far less often. Within the 
TMU, the division of specific MSP roles and responsibilities by the supervisory traffic management coordinator 
(STMC) among the traffic management coordinators (TMCs) remained flexible, with divisions by altitude strata, 
geographic area, and airspace problem (e.g, weather constraint or traffic volume) all possible contingent on the 
situation. On a more tactical scale, radar controllers and associates also found the MSP tools useful and effective 
for trial planning and coordinating clearances within their own and with other sectors. Other than voice 
communication with aircraft which was always performed by the radar controller, the division of other roles and 
responsibilities between radar controllers and associates varied by sector team and by equipage level. 

In summary, the MSP concept, as operationalized, prototyped, and tested in this simulation, appears both feasible 
and beneficial in a mixed equipage environment. 
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i Introduction 

Introduction of NextGen communication, navigation and air traffic management technologies - air-ground data 
communications and satellite-based navigation systems; new air traffic control and traffic management tools - 
provide an opportunity to move away from today's methods for controlling aircraft along fixed route structures, 
and towards more active management of en route aircraft trajectories. This shift towards trajectory-based 
operations will allow greater flexibility in modifying traffic flows to current conditions, which can improve both 
user and airspace efficiency. 1 2 If the tools and procedures for trajectory and flow management are sufficiently 
responsive, trajectory-based operations should also allow the air traffic system to respond more effectively to 
changes that occur within a 20-90 minute timeframe, thus becoming more robust to local disruptions such as 
weather or traffic congestion. 

This move away from sector-oriented air traffic control towards local area or flow-based trajectory management 
will require new tools and procedures for situation assessment and trajectory management, along with changes to 
roles and responsibilities within the en route facility team. For example, the current methods for trajectory 
modification available to traffic management are too limited, and the sector controller's geographic and temporal 
scope is too narrow, to support effective flow intervention in the proposed 20-90 minute timeframe. An 
operational framework for strategic trajectory clearance development from a non-controller station would satisfy 
this need. 

The multi-sector planner (MSP) concept explored in this simulation is a process for developing strategic trajectory 
clearances that address local flow management objectives. Decision support automation is used to identify and 
assess local area problems, and to solve them by modifying the trajectories of one or several aircraft. Solutions are 
coordinated as needed between traffic management and the operational area to make sure they are compatible 
with any existing constraints. MSP operations address situations that extend beyond the controller's planning 
horizon (e.g., downstream weather, traffic complexity, excess sector load), and are therefore developed from a 
non-controller planning position that can take a more strategic, "multi-sector" perspective. Proposed clearances 
are sent to the controller, using ground-ground data exchange capabilities, for review and delivery to the aircraft. 

1.1 Background 

The MSP concept evolved through a series of studies that explored the introduction of a new "multi-sector 
planner" position into the United States National Airspace System (NAS). 3 10 The multi-sector planner concept had 
been a topic of research in both Europe and the United States since the mid-1990s, with varying roles and 
responsibilities proposed for this new position. 11 24 In an FAA-sponsored study that was completed in 2006, two 
alternative concepts were developed and compared for feasibility and effectiveness. In one concept, the multi- 
sector planner worked from a remote location, acting as a single radar associate (or D-Side) controller who 
supported several sectors' operations. An alternative concept defined the MSP as a local area flow planner, 
responsible for managing sector load and complexity by selectively modifying trajectories or negotiating trajectory 
changes for aircraft. A human-in-the-loop (HITL) simulation was conducted in 2006 to compare these two concepts 
to a baseline condition that had radar associates for each sector and no multi-sector planner. 3 5 Results indicated 
that the area flow planner concept was preferred, and provided satisfactory support for sector operations. 

A follow-on research collaboration between NASA and the FAA began in 2007 to develop the area flow planner 
concept in the following areas: 

1. Defining roles and responsibilities for a new multi-sector planner position within the air navigation service 
provider (ANSP) team, along with corresponding changes to other team positions (traffic management 
coordinator (TMC), area supervisor and radar controller); 
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2. Developing communication and coordination procedures, and communication infrastructure 
requirements for multi-sector planning operations; 

3. Developing an integrated set of multi-sector planning decision support tools for situation assessment, 
trajectory/flow manipulation, and plan coordination. 

A 2009 HITL simulation tested this expanded concept for multi-sector planning operations, comparing test 
performance with and without the addition of a dedicated multi-sector planner position. This study found multi- 
sector planning to be feasible and effective in both conditions. Although improved performance in sector load 
management (both traffic count and complexity) was observed in the condition that added the new multi-sector 
planner position, this improvement was attributed to increased team size (thus better task load distribution), and 
not to the new position itself. The need for creating a new position was not conclusively demonstrated. The 
alternative, simpler solution of integrating multi-sector planning operations into the existing ANSP team appeared 
satisfactory and would likely be easier to integrate into individual facility operations. 6 10 

1.2 2010 Simulation 

The 2009 study investigated MSP operations in an environment where all aircraft were equipped with an air- 
ground data communications (Data Comm) capability. This simulation decision was based on the assumption that 
Data Comm equipped aircraft would be well represented within NAS in the NextGen Mid-Term (2018) timeframe, 
and thus a Data Comm requirement for entry into high performance en route airspace would be reasonable. With 
changes in the economy, however, the transition to DataComm seems likely to proceed more slowly, raising 
interest in assessing high altitude Mid-Term concepts for their suitability in a mixed equipage environment. 

While the 2009 simulation indicated that MSP operations showed promise in a full Data Comm environment, it was 
unclear whether Data Comm was required for MSP. The follow-on HITL simulation reported in this document was 
conducted in May 2010 to investigate MSP operations in a mixed equipage environment. Aircraft were categorized 
as "equipped" or "unequipped" based on the presence or absence of a Data Comm capability that enabled them to 
receive auto-loadable clearances and communications transfer messages from the ANSP. The purpose of this study 
was to determine the feasibility and possible benefits of introducing MSP in a mixed equipage context, where not 
all aircraft were Data Comm equipped. Conclusions from the study will inform FAA decisions about the timing, 
purpose, and appropriate operational environment for introduction of MSP operations. 

1.2.1 Objectives and Research Questions 

The study was conducted to test the feasibility and benefits of MSP operations in a mixed equipage environment, 
and to determine the tool and procedure enhancements needed to support those operations. The operational 
feasibility assessment addressed two related questions: (1) are MSP operations feasible for unequipped aircraft, 
and (2) are they feasible in a mixed equipage context. Similarly, two categories of potential benefits were explored: 
(1) system performance improvements associated with MSP at different equipage levels (e.g., airspace throughput 
and controller workload), and (2) the possibility of providing variable service for equipage through MSP operations. 
Tool requirements (for both planning and controller stations), as well as planning and coordination procedures - 
within facility (TMU-to-operational area) and within sector (R-to-D) - were two other topics addressed in the 
study. 


1.2.2 Approach 

The simulation was planned as a "downsized" one week follow-on to the 2009 study, using a reduced airspace and 
fewer participants, but similar traffic problems. Controller and planner tool capabilities were adapted for 
unequipped aircraft and for a mixed equipage environment, and procedures were modified to distribute planning 
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responsibilities between the operational area and the traffic management unit (TMU) without dedicated MSP 
positions. Radar associate positions were also staffed in this simulation because of the increased workload 
associated with unequipped aircraft. 

Simulation scenarios and operational procedures were developed to investigate MSP mixed equipage operations 
with respect to three specific operational objectives: 

1. Local area traffic count and complexity management, 

2. Convective weather contingency management, and 

3. Ability to provide aircraft differential service for equipage. 

The concept and procedures that were designed to support these three objectives are described in section 1.3. 

1.3 MSP3 Concept for Mixed Equipage Airspace 

The multi-sector planning concept investigated in this simulation is a tool-supported process for solving local area 
problems by selectively modifying the trajectories of one or more aircraft. Figure 1 describes the nominal event 
sequence for multi-sector planning to address a local situation such as convective weather or sector overload, and 
illustrates how it is coordinated between the TMU and the operational area teams. 

1.3.1 Roles and Responsibilities 

The specific responsibilities of the individual positions as illustrated in Figure 1 can be summarized as follows: 

• Area supervisors and traffic management monitor local traffic situations, identify problems and respond 
to external requests (e.g., traffic management initiatives or reroute requests from other facilities or the 
Command Center). 

• The TMC plans trajectory change clearance requests, coordinating with area supervisor and others. 
Depending on problem scope, the TMU team may further divide the task. For example, an STMC may 
coordinate the plan with one or more TMCs who will develop the actual trajectory reroutes. If the reroute 
occurs more than 30 minutes downstream, a trajectory clearance may be sent by the TMC directly to 
the Data Comm equipped aircraft, otherwise it is sent as a coordinated clearance (CC) request to the 
controller for review and execution. 

• The area supervisor manages plan execution by controllers. The area supervisor may also use planning 
tools to develop and coordinate within-area trajectory changes. 

• Controllers review CC requests and execute if suitable. 

1.3.2 Decision Support Tools 

Decision support tools are needed to support MSP operations in three areas: situation assessment, multi- 
trajectory trial planning, and coordination. 

Situation assessment (SA) tools enable traffic management and operational area supervisors to monitor local 
traffic, identify problems and evaluate solution options. Traffic flows, convective weather, sector load and 
complexity predictions are some of the monitored parameters. 

Multi-trajectory trial planning tools located at a (non-controller) planning workstation support the development of 
strategic trajectory-change clearances for one or several aircraft. Trial planning is integrated with the SA tools so 


MSP3_Report_3-2-ll 


3/2/2011 


10 


the planner can determine whether the changes meet flow management objectives, and assess their impact on 
local sectors. 

Coordination tools enable proposed plans and clearances to be shared using ground-ground data exchange 
automation. Data exchange functionality is integrated with the planning tools so that multi-trajectory trial plans 
can be sent to other planning stations or to the area supervisors for review. Finally, proposed "coordinated 
clearance" (CC) requests are sent from the planning station to the controller for review and execution, using 
ground-ground data exchange capabilities that are integrated with the controller's trial planning, conflict probe, 
and Data Comm automation. 




1. A problem is predicted for Area 1 in 45 minutes. TMU 
and Area 1’s supervisor discuss its impact and possible 
solutions. 


4. TMU then confirms with the Area 3 supervisor that the 
Area 3 controllers can execute plan. 



bound push that could be rerouted to reduce the impact 
on the affected sectors. 



5. TMU prepares and sends 3 coordinated clearance (CC) 
requests to Area 3’s sector A. Sector A’s controller 
reviews each request and sends the clearance to the 
aircraft if it is acceptable. 



3. The aircraft would be rerouted through Area 2. TMU 
confirms with Area 2 supervisor that this is acceptable. 
TMU may also check with Area 4. 



6. The Area 1 team manages the reduced flow through the 
two problem sectors. 


Figure 1. Nominal representation of multi-sector planning operations for local traffic management. 

1.3.3 Mixed Equipage Operations 

Trajectory planning operations are used to develop reroutes for both equipped and unequipped aircraft. 
Coordinated clearance requests for unequipped aircraft use a format suitable for voice delivery, thus the trajectory 
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clearance options are more limited (i.e., a clearance defined by a sequence of lat/long coordinates is not feasible 
with voice). Data Comm equipage also enables automated transfer of communication, reducing controller task 
load associated with sector entry and exit. These differences - reduced clearance options, voice requirement for 
clearance delivery and transfer of communications - mean that unequipped aircraft create more workload for 
controllers than equipped aircraft. Equipage-based differences also influence the planner's decision making 
regarding whom to reroute for sector load management, and timing and alternatives for convective weather 
management, as described below. 

A mixed equipage environment also provides a chance to offer service for equipage. This seems a reasonable 
additional objective for MSP operations: since the planning team does not have the same immediate separation 
and control responsibilities as the radar controller, they can focus on other issues when addressing local area 
problems. The radar controller continues to prioritize safety and local airspace efficiency. 

1.3.4 Sector Load Management 

Maintaining sector load within manageable limits represents both an MSP objective in its own right and a 
constraint on other flow management actions. Area supervisors and traffic management use SA tools to monitor 
the local traffic situation, particularly the predicted aircraft count and complexity for sectors of interest. Traffic 
management also monitors external situations that may have a local area impact. When solutions are developed, 
one constraint is that aircraft reroutes maintain or achieve an acceptable load distribution among sectors. 

Because of the increased controller workload associated with unequipped aircraft, a strategy that preferentially 
reroutes unequipped aircraft away from the problem area can be an effective way for planners to manage sector 
load. This strategy can result in fewer aircraft being rerouted overall, and has the added benefit of providing 
service to the equipped aircraft in the form of access to constrained airspace. 

1.3.5 Convective Weather Traffic Management 

MSP operations are used to develop reroutes for convective weather. Planners assign the unequipped aircraft 
strategic "playbook"-type routes that avoid both weather and equipped aircraft. The planner may also develop 
custom weather-avoidance trajectories for equipped aircraft. Trajectories for equipped aircraft can also be "fine- 
tuned" later by the controller if the situation is uncertain. This increased flexibility can provide better service, and 
is feasible because Data Comm supports the development and delivery of more complex solutions. 

1.3.6 Priority Service for Equipped Aircraft 

As described in the preceding sections, MSP complexity management and weather avoidance strategies naturally 
favor the Data Comm equipped aircraft, and trajectory-based solutions in general can be designed to provide 
priority service to equipped aircraft. This may, for example, involve providing them preferred access to constrained 
resources, e.g., rerouting unequipped aircraft instead of equipped aircraft to reduce sector load or complexity or 
giving an equipped aircraft priority access to a preferred weather avoidance route. Note that both of these choices 
can benefit system efficiency as well, since these aircraft require less effort from the controller and thus increase 
resource capacity. 

Controllers may also apply "service for equipage" conflict resolution strategies (i.e., move the unequipped aircraft 
first) on a workload-permitting basis, which they were encouraged to do during the simulation. 

1.3.7 Key Mixed Equipage Concept Decisions 

Many elements of the MSP concept features described above had alternative solutions. Some key choices that 
were made for the simulation, along with their rationale, include the following: 
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• Trajectory clearances and coordinated clearance requests would be provided for all aircraft. An alternative 
approach would have been to provide trajectories for equipped aircraft only. This would have given the 
planners less flexibility in solution development, and risked penalizing equipped aircraft since they were the 
only practical choice when rerouting was needed. 

• Graphical trial plan tools support trajectory development using named waypoints for unequipped aircraft. 
Named waypoints support voice clearance delivery and facilitate ground-ground verbal and data coordination, 
and a graphical trial plan interface might make this a more practical clearance for the controller to use. If so, it 
provides a means to introduce limited trajectory-based operations (TBO) for unequipped aircraft. 

• Controllers are encouraged to keep all aircraft on known trajectories. This involves issuing route clearances to 
unequipped aircraft when possible instead of heading clearances, and amending the ATC system entry as 
needed to maintain close conformance with the flown trajectory. While this adds to the controllers' workload, 
it improves the prediction accuracy of applications that rely on up-to-date trajectories (e.g., conflict 
detection). On balance, it seemed useful to explore the feasibility of maintaining trajectory conformance for 
unequipped flights. 

• Simulation operations support NextGen "service for equipage" objective . There were at least two alternatives 
that could have been adopted. With an "equipage blind" approach, planners and controllers might ignore 
aircraft equipage and choose the most expedient solution to a given problem. An "equipage neutral" approach 
could actively seek to insure that neither equipage type receives better (or worse) treatment. The "service for 
equipage" approach was selected because there was interest in exploring its feasibility (in terms of overall 
efficiency and impact on unequipped flights) and its compatibility with multi-sector planning operations. In 
addition, as mentioned above, a service for equipage approach can provide other system benefits such as 
increased system throughput and reduced controller workload. 

• Radar associate present to assist with added workload and complexity . The ratio of equipped to unequipped 
aircraft was a key independent variable in the simulation, with different scenarios having 10%, 50% or 90% 
equipage levels. A radar associate position was added so the same traffic levels used in the 2009 simulation 
could be maintained in spite of the increased sector complexity and workload, especially in the 10% equipage 
case. It also allowed participants to explore different ways of dividing the tasks when both controllers had 
equivalent tool sets, and to see how task distribution might vary with equipage level. 
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2 Methods 


2.1 Test Plan 

Overview A 1-week study was conducted to evaluate MSP operations in a mixed equipage environment. The 
week included two travel days, 1 day of training, and 2 days of data collection. Eight 60-minute data collection runs 
were completed. During each run, an ANSP team was presented with a combined convective weather and traffic 
load problem within four high altitude en route sectors and the surrounding airspace. 

The ANSP team included traffic management personnel, area supervisors and air traffic controllers. Six staff 
members from FAA air traffic facilities who currently work as operational area or traffic management supervisors 
participated in the study. Retired Oakland Center controllers staffed the remaining ANSP positions. 

A detailed description of the simulation airspace, staffed positions, scenarios and test matrix are provided below. 

Airspace. The test airspace (Figure 2) included four test sectors on the eastern end of Kansas City Center (ZKC), 
and was divided into two areas: ZKC-North (sectors 94 and 98) and ZKC-South (sectors 29 and 90). The traffic flow 
management problem, which extended into the surrounding ghost airspace, included convective weather in 
Indianapolis and Chicago Centers. The floor of the full simulation airspace - test sectors + ghost airspace - was 
FL290. 



Staffed positions. ANSP positions were staffed in both the TMU and the operational area. The ZKC traffic 
management team included two traffic management coordinators (TMCs) and one supervisory TMC (STMC). ZKC- 
North and ZKC-South each had an area supervisor and shared a "supervisor's assistant." Test sectors were staffed 
by a 2-person team consisting of a radar controller (R-Side) and a radar associate (D-Side). One TMC and three 
controllers managed the ghost airspace. Controller positions are shown in Figure 3. The supervisory and traffic 
management staff, who collectively comprise the multi-sector planning team, and have access to planning 
workstations, are shown in Figure 4. 

Six current FAA facility personnel were test participants. Two worked as ZKC area supervisors, two were radar 
controllers (sectors 29 and 94), and two alternated between ZKC TMC and STMC positions. The remaining positions 
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were staffed by simulation confederates. ANSP confederates were retired Oakland Center personnel. Pseudo-pilot 
confederates, who responded to ATC instructions, were either corporate pilots, general aviation pilots, or aviation 
students. 



ZKC Test Airspace Controllers: 

R 2 g R go f R 94 f R 9s : Test sector 
radar controllers 
D 29 D go D 94 D 98 : T est sector 
radar associates 

Ghost Airspace Controllers: 

R e : Radar East (high altitude) 

R w : Radar West (high altitude) 
R l : Radar, Low alt. (below FL290) 


Participants (white labels) 


Confederates (black labels) 



ZKC Test Airspace Planning Team: 
T kc : ZKCTMCs (2) 

ST kc : ZKC STMC 
S N : ZKC North area supervisor 
S s : ZKC South area supervisor 
sa: Supervisor assistant 

Ghost Planner Position: 

T g : Ghost TMC (performs TMC 
and STMC duties for all other 
facilities). 


Participants (white labels) 


Confederates (black labels) 


Figure 4. "Planning" positions (traffic management and area supervisors). 


Test matrix and test scenarios. The test matrix combined 2 convective weather patterns ( W1 and 1/1/2), 2 
traffic patterns ( Traffic 1 and Traffic 2), and 2 scenario variants (A and 6), for a total of 8 unique traffic/weather 
combinations (Table 1). 

Each traffic pattern was divided into two sets of aircraft: Traffic 1 was split into two unequal sets with a 10/90 size 
ratio, while Traffic 2 had two sets that were equal in size (a 50/50 ratio). 


Table 1: 8 Traffic + Weather Combinations 



10/90 Mix 

50/50 Mix 

W1 

1A 

IB 

2A 

2B 

W2 

1A 

IB 

2A 

2B 
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Within a run, each set of aircraft were designated as Data Comm equipped or unequipped. Scenario variants A and 
B alternated aircraft equipage assignments within a particular traffic pattern: the equipped aircraft set in A became 
the unequipped set in B, and vice versa. Combining traffic patterns with scenario variants resulted in three 
different equipage levels, with 10%, 50% or 90% equipped aircraft. Each scenario presented a combined convective 
weather and sector overload problem for the planning team to solve, with convective weather located within and 
downstream from the four ZKC test sectors. 

Figure 5 illustrates a sample problem. In this situation, the planner may reroute some aircraft for weather 
avoidance and/or complexity reduction in the test sectors affected by convective weather entering from the north 
(e.g., dashed purple and green route adjustments). Downstream weather can further complicate the problem. The 
blue flow can be rerouted north or south of the downstream weather, but the southern reroute reduces 
congestion in sector 98. The planner's decisions may also vary based on aircraft equipage type. For instance, the 
planner may choose to leave equipped aircraft in the purple flow untouched and allow the sector controllers to 
adjust the routes later if needed. 

Although the traffic within each of the two A/B sets (1A and IB, 2A and 2B) was identical, swapping the equipage 
assignments within each traffic pattern presented markedly different problems for the planning team, since the 
strategies for handling equipped and unequipped aircraft were different. Controllers would see further differences 
between the A/B variants because of upstream actions taken by the planners before the aircraft entered the test 
sectors. 

Swapping equipage within Traffic 2, the 50/50 mix, enabled analysis of service for equipage strategies, since the 
equipage ratio is the same, but each aircraft's equipage assignment is changed. Swapping equipage for the 10/90 
mix resulted in two different equipage levels (10% equipped and 90% equipped), and allowed investigation of the 
impact of high vs. low equipage levels on feasibility and performance. 

The final test run sequence is shown in Table 2; with each scenario built from a unique combination of 2 traffic x 2 
equipage assignment alternatives x 2 weather patterns. The resulting equipage levels are listed. 



Figure 5. Example of test scenario weather and traffic interactions 
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Table 2: Final Test Run Execution Sequence 


Run 

Scenario 

Equipage Level 

Comments 

1 

T1B_W1_90 

90% equipped 


2 

T2BW250 

50% equipped 


3 

T1A_W1_10 

10% equipped 

discarded run 

4 

T2A_W2_50 

50% equipped 


5 

T2B_W1_50 

50% equipped 


6 

T1A_W2_10 

10% equipped 


7 

T2A_W1_50 

50% equipped 


8 

T1A_W1_10 

10% equipped 

repeat of run #3 

9 

T1B_W2_90 

90% equipped 



2.2 Simulation Environment 

MSP tools and procedures that were developed for the 2009 simulation assumed that all aircraft were Data Comm 
equipped. 8 Tools and procedures were modified for the 2010 mixed equipage simulation in two areas: (1) MSP and 
controller tools were adapted for unequipped aircraft and mixed equipage traffic; and (2) controller tools and 
procedures were modified so that a radar associate could assist with sector operations. Refer to NASA, 2010 s for a 
more extensive description of the MSP tools. 

Planning and controller workstations are illustrated in Figure 6 and Figure 8. As these figures indicate, both MSP 
and controller tools were presented within an emulated DSR framework. The DSR convention for mapping input 
command entries to a two-character identifier - e.g., "QZ [altitude] [aircraft id]"for changing an aircraft's assigned 
altitude - was maintained. Existing DSR input codes were supplemented by new two-character commands for the 
simulated NextGen functions. Training material for controllers and planners that lists available input commands 
and their syntax is provided in Appendix A. 

2.2.1 MSP Tools for Mixed Equipage Operations 

MSP automation, as described in section 1.3, includes decision support tools for situation assessment, multi-sector 
trial planning and ground-ground coordination. Figure 6 shows a planning station from the 2009 simulation that 
has an integrated set of these capabilities. An overview of the MSP tool set is provided in this section, with a 
description of the enhancements that were made to support mixed equipage operations. 

Situation assessment tools. A Traffic Situation Display (TSD) provides a real-time display of current traffic for 
the local facility and its neighbors, with color-coded flows and a weather loop showing recent weather radar 
images and the weather's projected future movement. A weather penetration probe identifies aircraft that are 
predicted to penetrate convective weather at three selectable intensity levels. A modified en route DSR radar 
display shows a multi-sector view of the current traffic, and a set of dynamic, interactive filters enable the planner 
to selectively highlight the data blocks for subsets of this traffic. These filters allow specific flows or sets of aircraft 
to be identified by flight level, departure or arrival airport, waypoint, equipage type or predicted weather 
penetration. 


The DSR ("display system replacement") workstation is used today by radar controllers in NAS en route facilities. 
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Interactive 
load table 


TSD with weather loop 


DSR display with: 
Weather penetration probe; 
Multi AC trajectory planning; 
Automatic trajectory generation for 
conflict and Wx avoidance 
(available on all positions) 


Interactive load graphs 


Dynamic filter view: 

To/From, VIA, FL, Draw, WX, Conflict, 
and many more tailored by each individual 


Load control display 
window (for graph/table 
complexity parameters) 


Figure 6. Multi-Sector Planning Tools 


The planner also has an interactive sector load table and load graphs (Figure 7). The load table displays predicted 
values for individual sectors - complexity, aircraft count, weather penetration events and/or other measures - in 
15 minute increments, and load graphs show predicted sector values in 1-minute increments. These displays can 
be used to actively filter the presentation on the DSR display: clicking on a load table entry or time slice in the load 
graph highlights the set of aircraft that contribute to that value. 



Figure 7: Sector Load Table (left), and Sector Load Graphs (right). 
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The complexity calculation incorporates specific attributes of the traffic and can provide a useful metric for 
predicting and managing controller task load. A variety of parameters have been proposed and tested as inputs for 
calculating complexity (number of arrivals, climbing or descending aircraft, conflicts, etc.). 25 In order to support the 
sector load management task, an effort was made to select intuitive input parameters that could be actively 
managed by the planning team. A key contributor to controller workload in this simulation was the number of 
unequipped aircraft: as mentioned earlier, the controller's communication, handoff and monitoring tasks were 
much easier for equipped aircraft. This was addressed in the complexity computation by using a differential 
weighting for equipped and unequipped aircraft. 

Multi-aircraft trial planning tools. The planning station also has multi-trajectory trial planning automation 
that supports the development of clearances and trajectory changes for one or several aircraft at a time. Filter 
tools can be used to highlight aircraft of interest, and trial plan routes can be developed using a click and drag 
function to insert or remove waypoints. Trial planning is integrated with the SA tools so the planner can determine 
whether the changes meet flow management objectives - e.g., weather avoidance, sector load redistribution, 
complexity reduction - without any unintended side-effects. 

Trajectories for Data Comm equipped aircraft are constructed using a graphical tool to click-and-drag the route 
and insert user-defined waypoints at precise locations. These waypoints are defined by lat/long coordinates, which 
are practical for Data Comm, but inappropriate for voice clearances. In order to support trajectory development 
for unequipped aircraft, a "snap-to" function was added. This function finds a named waypoint close to the desired 
trajectory change point, then substitutes that waypoint to construct a modified route that can be issued as a voice 
clearance to unequipped aircraft. 

Ground-ground coordination. Plan coordination and coordinated clearances are accomplished using radio and 
digital ground-ground data exchange. An emulation of the en route facility's voice-switching and communication 
system (VSCS) provides the mechanism for voice communication between the TMU, operational area, and other 
facilities. Ground-ground data exchange functionality is integrated with the planning tools. The coordinated plan 
(CP) command can be used to send multi-trajectory trial plans to sender-specified planning stations or area 
supervisors for review and discussion. Proposed coordinated clearance requests (CC command) are sent from the 
planning station to the sector that currently controls that aircraft for the controller to review and execute. 

2.2.2 Controller Tools for Mixed Equipage Operations 

The controller station combined a DSR emulation with additional automation tools as shown in Figure 8. Additions 
included NextGen tools such as trajectory-based conflict and weather probes, a Data Comm interface, and route 
and altitude trial planning. Ground-to-ground data exchange automation for receiving and sending CC requests 
supported multi-sector planning as well as controller-to-controller coordination. This feature was fully integrated 
with the controller's trial planning, conflict probe, and Data Comm automation, which supported the sender in 
developing acceptable clearances, and the receiver in quickly evaluating their operational suitability and issuing 
the requested clearance. This functional integration was critical to the operational feasibility of MSP operations. 

A controller D-Side station that had the same automation tools as the R-Side was also used in this simulation. 

Some D and R behaviors could be selectively coupled between stations (e.g., data block movement). How the 
stations were configured, and the role of the radar associate was a decision made by the radar controller. 
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Interactive FDBs with 
route and altitude trial 
planning, conflict probe, 
weather probe, and fly- 
out data comm windows 


D-side station with the same 
tool-set to assist R-side 


Datalink CC 
message under 
review (pink line) 


Interactive 
conflict list and 
datalink status list 


Mixed equipage environment: 
green FDB = equipped 
gold FDB = unequipped 


VSCS emulation 
for voice comm 


Figure 8. Controller Tools 


Mixed equipage features. Elements of the controller's tools were modified to support mixed equipage 
operations, including equipage-based flight data block differences (color, symbols, and content) and features of 
the trial plan and coordinated clearance interface. Flight data blocks for equipped and unequipped aircraft were 
assigned different colors to facilitate quick recognition of aircraft equipage type. As Figure 9 illustrates, this not 
only helped controllers categorize individual aircraft, but also enabled controllers and others to recognize their 
relative proportions and distribution within a sector or region. 
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Figure 9. Controller display with color-coded data blocks: equipped, green; unequipped, gold. 
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Figure 9 also illustrates the chevron-like target symbols used to indicate aircraft position and heading in this 
simulation environment, and the enlarged target that indicated when an aircraft was out of conformance with its 
assigned trajectory, or "free-track" (gold target in the center of Figure 9, left). Because of the increased 
effectiveness of decision support tools with TBO, controllers were asked to try to keep unequipped aircraft on 
assigned trajectories rather than using vectors for lateral clearances. The enlarged target was a feature that 
assisted controllers in monitoring aircraft status when vectors were in use, or when a flown trajectory was not in 
precise conformance with its representation within the ATC system. 


Figure 10 illustrates two coordinated clearance requests that were received by the controller: one for an equipped 
aircraft (left) and another for an unequipped aircraft (right). Incoming CC requests are indicated in the full data 
block (FDB) by a pink box background for the trial plan portal (arrow symbol) to the right of the aircraft's callsign. 
When the controller clicks on the communications symbol to the left of the callsign (diamond for equipped; dash 
for unequipped), the CC request is shown as a pink trajectory. As mentioned earlier, trajectory clearances for 
equipped aircraft are defined with lat/long coordinates, while unequipped aircraft trajectories use named 
waypoints, with new waypoints shown in the fourth line of the FDB. 



Figure 10. Pink coordinated clearances for equipped aircraft (left) and unequipped aircraft (right). 

Trial plan graphics for equipped and unequipped aircraft are shown in Figure 11. The routes are constructed as 
described earlier for multi-aircraft trajectory planning, with the equipped aircraft's route defined by lat/long 
coordinates, and the unequipped aircraft's route by named waypoints. Figure 12 illustrates these two different 
behaviors, showing the coordinates for AAL302's modified capture waypoint (i.e., its next active along path 
waypoint) and the sequence of waypoints - QJN, KRAZO - that comprised N304's route change. 



Figure 11. Cyan trial plan graphics for equipped aircraft (above) and unequipped aircraft (below). 


MSP3_Report_3-2-ll 


3/2/2011 


21 





Equipped aircraft clearances could be sent directly to the flight management system (FMS) and the ATC computer 
system at the same time by selecting an option in the Communication fly-out menu. Unequipped aircraft 
clearances were issued by voice, and the route was amended in the ATC computer with the Communication fly-out 
menu (Figure 12). 
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Figure 12. Communication Fly-out Menu: equipped aircraft (left), unequipped aircraft (right). 


Summary ofUC, QC, CC, and CP commands. The UC (uplink clearance) command sends a trial plan or 
coordinated clearance to a Data Comm equipped aircraft, and concurrently amends the ATC system data to match 
the clearance that was sent. A QC (amend ATC system entry) command also amends the ATC system data with a 
trial plan or coordinated clearance request, and is used with a voice clearance (usually to an unequipped aircraft), 
or when a system trajectory needs to be aligned with the aircraft's actual trajectory. The CC (coordinated 
clearance) command is used to send proposed clearances to the sector that currently has track control of the 
relevant aircraft. It can be used for multi-sector coordinated clearances (TMC or supervisor to sector controller), 
inter-sector coordinated clearances (sector controller to sector controller), and within sector coordinated 
clearances (radar associate to/from radar controller) for both equipped and unequipped aircraft. The CP 
(coordinated plan) command is used by planners to send sets of CCs to one or more other planning stations to 
support coordinated development and execution of a multi-sector plan. 

2.3 Experimental Procedures 

Simulation layout. The simulation environment was organized with the two small ZKC areas (North and South) 
and the ZKC TMU in three separate rooms. Pseudo-pilots and ghost positions were located in a fourth room. The 
complete lab layout is shown in Figure 13. The two ZKC TMU participants alternated between STMC and TMC 
positions. All other positions (supervisors North and South; radar controllers for sectors 29 and 94) worked the 
same positions throughout the simulation. 

Sequence of events. Each run began with a traffic management teleconference between the ZKC TMU, 

Command Center, and adjacent facilities. A confederate acting as a Command Center representative led a 
discussion about the status of current playbook routes, convective weather and other concerns, and provided a 
high level plan for modifying traffic flows to deal with the current situation. 

The STMC then organized the response within ZKC. The TMCs' activities were coordinated by assigning each a 
particular subset of the traffic problem - e.g., internal vs. external weather avoidance, northern vs. southern flows, 
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different altitude strata. The STMC also briefed the area supervisors about the plan, including particular reroutes 
that their controllers might be asked to implement. 

TMCs then developed specific reroutes to accomplish their assigned tasks and to maintain sector loads at 
manageable levels. The STMC monitored task execution, and airspace status, occasionally visiting the operational 
area to insure that the plan was working out. The area supervisor would brief his controllers about what to expect, 
monitor their task load and the developing situation, and keep the TMU informed as needed about area status. 
Finally, the radar controller determined the within-sector distribution of tasks with his D-Side, a split that might 
vary with weather and equipage level. 



Figure 13. Simulation positions and their physical locations 

Roles and responsibilities. Appendix B includes several slides from the two briefings that were provided to the 
simulation participants on the first day of the study. A slide from the introductory morning briefing provides a high 
level summary of the mixed equipage MSP concept. Following an initial session in the simulation laboratory, a 
second briefing provided a more detailed description of the roles and responsibilities and tasks for each position. 
Guidelines for providing service for equipage, and for division of responsibilities between radar controller and 
radar associate were covered in this presentation. 
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3 Results 


3.1 Data Overview 

MSP operations in a mixed equipage environment were evaluated to determine concept feasibility, in terms of 
safety, operator acceptability and workload; and benefits, with a focus on flight efficiency, system efficiency, 
airspace throughput and workload. Both objective and subjective measures were used in this evaluation. 

3.1.1 Objective Metrics 

Several different data logs were automatically generated during each run by different software processes within 
the simulation. Aircraft location and status parameters were recorded at regular intervals, for example, along with 
other data (e.g., current and predicted sector count, traffic complexity, number of conflicts). Time-stamped events 
(such as operator input actions, handoffs, weather penetration events or aircraft separation errors) were logged 
when they occurred, along with their relevant attributes. Objective measures derived from these data logs were 
used to evaluate efficiency and safety, and communication processes. 

Camtasia recordings. A continuous screen capture of each test position was made using the program Camtasia. 
The recordings were used to review the traffic situation and operator actions surrounding particular events. 

3.1.2 Subjective Metrics 

There were three sources of subjective data. Operators at all test positions were prompted to enter their within- 
run workload, on a scale of 1-6, at five minute intervals throughout each run. They also completed short on-line 
questionnaires after each run, and a longer questionnaire after the simulation. A post-simulation debrief discussion 
between operators and researchers was a final source of subjective data. These data supported the analysis of 
operational acceptability, workload, task distribution, and overall concept and tool performance. 

3.1.3 Methods & Selection Criteria 

This section explains some of the data reduction and recording methods, selection criteria, and terminology used 
in the analyses. 

Test positions. Event logs and subjective data were collected from all ZKC TMU and operational area positions, 
which were staffed by both FAA participants and retired confederates. 

Clearance and coordinated clearance event records. Several two-character commands associated with 
clearance entry and coordination were logged at each test position during data collection runs. Logged entries for 
three commands - UC (uplink clearance), QC (amend clearance entry in ATC system), and CC (coordinate 
clearance) - are analyzed below. Each time-stamped entry includes the position where it was entered, clearance 
content, and aircraft ID. UC and CC outcomes (accepted, rejected or no response) were also recorded. 

“Open loop” data. Each traffic scenario consisted of a set of simulated aircraft, where each aircraft had a 
controlled entry position, entry time and default flight plan. The scenarios played out differently during different 
runs based on planner and controller decisions that were made. For example, sector complexity and demand were 
modified by planner actions to alleviate load problems, and conflict and weather penetration counts were reduced 
because of controller actions. In order to get a picture of the underlying problem that was presented by each 
scenario absent operator input, data logs were recorded while each scenario was allowed to run open loop - i.e., 
with each aircraft flying its scripted default path with no operator intervention. 
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Test airspace vs. surrounding airspace. Aircraft trajectories were altered by ANSP actions taken inside and 
outside the test airspace. TMCs usually solved weather and sector load problems by rerouting aircraft before they 
entered the test sectors, occasionally routing them around the test airspace entirely to solve load or complexity 
problems. After an aircraft entered a test sector its trajectory could also be modified by the controller to resolve 
conflicts or avoid weather. Ghost controllers were responsible for managing handoffs and responding to external 
CC requests, but not for weather avoidance or conflict resolution within their airspace. Because of these 
differences, some measures (e.g., conflict count) were only meaningful when they occurred within the test 
airspace while other measures (e.g., path length) provided useful data in both regions. 

Discarded run. Run 3 was repeated due to simulation difficulties. Data from the original run were discarded. Of 
nine experimental runs, then, eight were analyzed. 

3.2 Efficiency Analysis 

Two metrics were used to investigate the MSP test operations with respect to system and user efficiency. A third 
measure - the equipage type of maneuvered aircraft when controllers resolved mixed equipage conflicts - 
explores a related user benefit mechanism and is also presented in this section. 

Test sector throughput was defined as the number of unique aircraft that were observed in a test sector during 
each 60 minute run. Test sector throughput varied as the planning team routed aircraft around the test sectors to 
manage load and complexity. Sector throughput by equipage type (i.e., throughput of equipped or unequipped 
aircraft only) reflected the planning strategies for managing demand, and showed how airspace access varied by 
equipage. 

Path length change - the path length difference between the original trajectory and the actual flown trajectory - 
served as a measure of user efficiency, and allowed the impact on individual aircraft of planner and controller 
actions (and of the tools for performing those actions) to be evaluated. The original (flight plan) path length, 
measured from simulation entry location to destination, was compared to the sum of the actual distance flown 
plus remaining distance to destination. The difference between these two values provided a path length delta for 
each aircraft that was caused by actions taken by controllers or planners during the simulation. Lateral, but not 
vertical, distances were used in the calculation. 

In general, overall throughput increased as equipage level increased. Throughput was also higher for equipped 
than for unequipped aircraft, indicating that MSP operations were successfully used by the TMU to selectively 
reroute the unequipped aircraft away from overloaded or weather-constrained airspace. Higher test sector 
throughput for equipped aircraft also suggests the MSP function in the TMU can be an effective means to provide 
service for equipage. Path length changes showed a benefit for equipped aircraft, with smaller changes observed 
on their flown distance when compared to the unequipped aircraft. This difference was consistent at all equipage 
levels, and it was observed both for aircraft that flew through the test airspace and those that flew around it. 

Resolution of mixed equipage conflicts. Radar controllers and associates were asked to give priority service, 
when able, to equipped aircraft in mixed equipage conflicts (conflicts between an equipped and an unequipped 
aircraft) by maneuvering the unequipped aircraft and leaving the equipped aircraft on its current trajectory. In the 
vast majority of cases (108 out of 132) the test controllers' resolution maneuvers moved the unequipped aircraft, 
even when that solution required coordination with another controller. 

3.2.1 Test Sector Throughput 

Observed changes in mean test sector throughput as a function of equipage level are shown in Figure 14. Total 
sector throughput increased when more aircraft were equipped. A one-way ANOVA performed on the mean sector 
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throughput for the four ZKC test sectors revealed that a significant difference by condition (F(2,5) = 45.93, p < .01), 
and post hoc tests revealed the mean sector throughput in the 10% equipage condition was significantly lower 
than in either the 50% or the 90% conditions (Tukey's HSD ps < .05; Figure 14). The observed difference between 
the 50% and 90% conditions did not reach significance (Tukey's HSD p > .05). 
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Figure 14. Average sector throughput per run, by equipage level. 

The cumulative throughput of the four test sectors was significantly higher for equipped aircraft than for 
unequipped aircraft (x 2 i = 17.24, p < .001; Figure 15). As Figure 15 shows, there were 250 more equipped than 
unequipped aircraft that were allowed to fly through the test sectors when combined across all 8 runs. 



Figure 15. Cumulative throughput across all test sectors and all runs, by equipage type. 

A consistent airspace access advantage was observed for individual aircraft when they were equipped compared to 
when they were unequipped (Table 3). Priority service for equipped aircraft was seen when comparing aircraft set 
outcomes between the A and B traffic scenario pairs. The difference was significant for both traffic scenarios, 
although only marginally so for the 50/50 mix, Traffic 2 (x 2 iS = 15.84 and 3.65, ps < .001 and .06, respectively). 


Table 3. Cumulative throughput totals for sets of aircraft when unequipped and equipped. 


Traffic Scenario Pairs 

Runs 

Unequipped 

Throughput 

Equipped 

Throughput 

Advantage to 
Equipped 

1A (10% Eq.) and IB (90% Eq.) 

1, 6, 8, 9 

807 

975 

168 

2A and 2B (both 50% Eq.) 

2, 4, 5, 7 

881 

963 

82 
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3.2.2 Flight Distance 

All simulation aircraft. Mean path length difference was computed for all aircraft that flew in the simulation at 
FL290 and above. Most aircraft saw no change, some had their path length reduced, and some increased; averaged 
over all aircraft, however, path length was increased as participants rerouted aircraft to solve the traffic and 
weather problems in the simulation airspace. The average per-aircraft increase was greater for unequipped aircraft 
than for equipped aircraft (F(l,7963) = 22.09, p < .001; Figure 16). The mean path length increase for unequipped 
aircraft (M = 1.91 nm) was over 2.5 times that of equipped aircraft (M = 0.69 nm). This was consistent both with 
providing priority service to the equipped aircraft and with a complexity management approach that moves the 
unequipped aircraft around congested airspace. 
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Figure 16. Mean path length change for all aircraft in the simulation, by equipage type. 

Path length impact inside and outside test sectors. When aircraft were sorted according to whether their 
path length was increased, decreased or unchanged (path length delta less than 1 nm), the distribution differed 
between aircraft that transited the test airspace (whose trajectories could have been modified by planners and/or 
controllers) and those that did not (subject to planner modifications only). For aircraft transiting the test airspace, 
proportionally the same numbers of unequipped and equipped aircraft incurred path length decreases, increases, 
and no path length change (Ts: Decreased = 185, 182, Increased = 344, 387, No change = 572, 647, respectively, Xi 
= 1.46, p > .05; Figure 17, left). 


Figure 17. Number of aircraft whose path length was decreased, increased or unchanged. 
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For aircraft that never entered the test airspace, proportionately more unequipped than equipped incurred path 
length increases (Ts: Decreased = 207, 173, Increased = 328, 234, No change = 2341, 2364, x 2 = 16.93, p < .001; 
Figure 17, right). This is consistent with the observed throughput difference, and with the assumption that 
planners routed more unequipped aircraft around the test airspace, since most of these deviations would result in 
path length increases. 

Although the majority of aircraft that transited the test sectors saw no change in their path length, the average for 
all aircraft was a modest increase, with the unequipped aircraft incurring a significantly greater path length 
increase than the equipped aircraft. The same pattern was also observed for aircraft that did not transit the test 
airspace (F(l,2315) = 10.61, p < .001, Figure 18, left; F(l,5645) = 13.86, p < .001, Figure 18, right). When comparing 
change magnitude (Figure 18), the unequipped aircraft whose flight path was changed saw a greater average 
increase (~9 nm), and a greater average reduction ( ~2 nm), than the equipped aircraft. These larger impacts are 
probably due to the difference in how the reroute trajectories are defined. Data Comm equipage permitted use of 
lat/long coordinates, thus tailoring of more precise lateral trajectories, while unequipped aircraft trajectories were 
limited to the sparse network of named waypoints available for trial planning. 
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Figure 18. Mean change for A/C with decreased path length, increased path length, and all aircraft. 

We were also interested in whether the path length advantage for equipped aircraft held across equipage levels. 
Because our traffic scenarios most accurately achieved our target equipage levels (10, 50, and 90%) within the test 
area, we compared change in path length across equipage levels only for those aircraft that transited a test sector. 
Once again, the average path length increase per aircraft was greater for unequipped than for equipped aircraft, 
and the advantage for equipped aircraft held across all three equipage levels (Figure 19). A repeated measures 
ANOVA with within-run variable equipage type (unequipped or equipped) and between-run variable equipage 
level (10%, 50%, or 90%) confirmed a main effect of equipage type (F(l,5) = 7.16, p < .05), but no main or 
interaction effects of equipage level (F( 2,5) = 0.02, p > .05 and F(2,5) = 0.00, p > .05, respectively). On average, the 
increase in path length for unequipped aircraft transiting a test sector (M = 3.8 nm) was over twice that for 
equipped aircraft (M = 1.9 nm) (Figure 19). 
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Figure 19. Mean path length change per aircraft by equipage level, for aircraft transiting the test airspace. 


3.2.3 Service for Equipage in Conflict Resolution 

Radar controllers and associates were asked to give priority service, when able, to equipped aircraft in mixed 
equipage conflicts (conflicts between an equipped and an unequipped aircraft) by maneuvering the unequipped 
aircraft and leaving the equipped aircraft on its current trajectory. 

The average number of mixed equipage conflicts per run for each equipage level is shown in Figure 20. 
Unsurprisingly, the number of mixed equipage conflicts was greatest in the 50% condition; a one-way ANOVA 
revealed a significant difference between conditions (F( 2,5) = 45.93, p < .01), and post hoc tests revealed the mean 
number of conflicts in the 50% condition was significantly higher than in the 10% and than in the 90% condition 
(Tukey's HSD ps < .01; Figure 20). There was no significant difference between the 10 and 90% conditions in the 
number of mixed equipage conflicts (Tukey's HSD p > .05). 
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Figure 20. Mean number of mixed equipage conflicts per run, by equipage level. 


Across the eight runs, there were a total of 167 mixed equipage conflicts, 131 of which were solved by controller 
actions. The equipage of maneuvered aircraft were identified for each resolved conflict. Twenty-nine of the 
remaining conflicts disappeared with no controller action (probable false alarms); four resulted in separation 
violations; and three occurred at the end of a run and were still unresolved when the run ended. 
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More unequipped than equipped aircraft were maneuvered to resolve mixed equipage conflicts. A repeated 
measures ANOVA with within-run variable of equipage type of maneuvered aircraft (unequipped or equipped) and 
between-run variable airspace equipage level (10%, 50%, or 90%) revealed main effects of both equipage type and 
equipage level (F(l,5) = 22.13, p < .01 and F(2,5) = 58.79, p < .001, respectively), which were qualified by a 
significant equipage type by level interaction (F( 2,5) = 10.08, p < .05). An inspection of the means revealed that 
unequipped aircraft were moved more often than equipped aircraft and that this difference was more pronounced 
in the 50% condition (Figure 21). 

In 23 of the 131 mixed equipage conflicts, the controller chose to move the equipped aircraft. This usually occurred 
when the unequipped aircraft was off the radar scope and/or not controlled or "owned" by the controller resolving 
the conflict. In some cases, the equipped aircraft was changing or about to change altitude (e.g., arriving or 
departing) at the time, and the controller chose to use a vertical resolution with the transitioning rather than 
moving an unequipped aircraft in level flight. In other cases, however, when the conflict was imminent (less than 
1-2 minutes to separation violation), it appears controllers may have found the relative ease of moving the 
equipped aircraft to be more expedient, since no voice clearance was necessary. This preference for using Data 
Comm when a fast response was needed was likely a simulation artifact. The reverse preference would be 
expected in an operational setting, where voice communication gives the controller immediate feedback that the 
clearance was heard and will be executed. 



Figure 21. Mean number of A/C of each type maneuvered to resolve mixed equipage conflicts per run. 
3.3 Clearance and Coordinated Clearance Analysis 
3.3.1 Ground-Ground Coordinated Clearance Requests 

Across the 8 runs, a total of 1595 ground-ground CCs were recorded. Nearly twice as many CCs were sent for 
unequipped aircraft than for equipped aircraft (T unequipped = 1026, T equipped = 569). This reflects a planning strategy 
for preferentially moving unequipped aircraft, and also indicates that participants found these tools to be usable 
and effective for planning trajectory modifications for unequipped aircraft. Out of the 1595 recorded CCs, 229 
went to the four test sectors, and 1366 to the confederate controlled ghost airspace (Table 4). 
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Table 4. Number of CC requests sent to test sectors and ghost sectors, by equipage type. 


Recipient 

Unequipped 

Equipped 

Total 

ZKC test sector 

202 

27 

229 

Ghost sector 

824 

542 

1366 

TOTAL 

1026 

569 

1595 


Most of the CC requests received by "Ghost" sectors were sent from a within-facility planning station -TMC or 
supervisor. Most CCs received by the a test sector were sent by another controller (166 out of 229; see Table 5). 
These were either for within-sector (R-D) coordinated clearances or for between-sector coordination of conflict 
resolution or weather avoidance maneuvers. These results indicate that the controllers in particular found data 
exchange useful for coordinating clearances for unequipped aircraft. Unlike the planners, controllers had other 
options for within (and between) sector coordinated clearances. The fact that they chose to use the CC function so 
frequently - particularly for within-sector coordination - suggests that they found it both useful and usable. 


Table 5. Number of CC requests sent to test sector and ghost sectors, by sender and receiver. 


Sender: 

Receiver: 

ZKC TMCs 

Ghost TMC 

ZKC Sups 

Test sector 
(RorD) 

TOTAL 

ZKC test sector 

35 

3 

25 

166 

229 

Ghost sector 

546 

732 

19 

69 

1366 

TOTAL 

581 

735 

44 

235 

1595 


Coordinated Clearance request outcomes. Out of the 1366 CCs sent to the ghost sectors, 1356 were 
executed, 0 were rejected, and 10 were 'ignored' - i.e., no response was sent back to the sender. Nine of these 
ignored requests were sent within 3 minutes of the end of the run, and the 10 th was a descent request sent for an 
aircraft that was already in descent. 

Out of the 229 CCs sent to the test sectors, 218 were executed, 10 were rejected, and 1 was ignored. The ignored 
request to sector 98 was sent 14 seconds before the end of the run. All ten rejected requests were for unequipped 
aircraft, with 9 sent by the D-Side controller to his/her own R-Side, and 1 from TMC Ghost to sector 90. 

Of the nine D-side initiated requests that were rejected, five were actually cancelled by the D-side himself: twice 
because the problem went away (one conflict, one predicted weather penetration), and three times to replace it 
with a better solution. Two of the four rejects from the R-side were for weather problems that proved 
unnecessary, one conflict resolution was improved (a lat/long replaced by a named waypoint), and one may have 
been an error, since the clearance was issued anyway. 

3.3.2 Air-Ground (Data Comm and Voice) Clearances 

Although proportionally the same number of unequipped as equipped aircraft received clearances in the four ZKC 
test sectors, the unequipped aircraft represented significantly more work for controllers. The per run mean 
percentage of aircraft of each type that received at least one clearance did not differ by equipage (M unequi p ped = 
36.9%, M equipped = 34.2%; paired t[ 7) = 1.06, p > .05). However, the mean number of controller input events per 
aircraft was greater for unequipped than equipped aircraft (M unequipped = 1.9, M equipped = 1.5; paired t( 7) = 5.32, p < 
.001; Figure 22). 
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Note that while all uplink clearances have an associated UC event record, there is not a one-to-one 
correspondence between logged QC events and clearances issued to the aircraft. For example, every tactical 
heading clearance may not have a corresponding QC entry; the controller may choose to wait until the aircraft is 
back on course before updating the system entry. Alternatively, a QC route amendment can be used to bring the 
system into tighter conformance with the actual route of flight even if no new clearance is issued. Even though the 
QC is an unsatisfactory indicator of the number of clearances that are issued, however, UC and QC counts are 
reasonable measures of controller task load for data maintenance associated with equipped and unequipped 
aircraft since they do reflect the number of computer input actions controllers took for each type of aircraft. If 
voice clearances were added to this analysis, the observed task load difference between equipped and unequipped 
aircraft would be even greater. 
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Figure 22. Mean number of clearance entry commands t per aircraft, aircraft that received at least one clearance. 

3.3.3 Controller Use of Coordinated Clearance Requests 

This section describes use of the CC function at the two participant staffed sectors, 29 and 94. A total of 107 
coordinated clearance requests were sent by radar and radar associate controllers working these two sectors, and 
89 of the 107 requests were for unequipped aircraft. A breakdown of who sent these CCs, by aircraft equipage, is 
presented in Tables 6 and 7. 


Table 6. Internal CCs (Sent to Own Sector) 


Sender 

Aircraft Type 

Total 

Uneq. 

Eq. 

29D 

24 

0 

24 

29R 

0 

1 

1 

94D 

26 

0 

26 

94R 

1 

2 

3 

Total 

51 

3 

54 


Table 7. External CCs (Sent to a Different Sector) 


Sender 

Aircraft Type 

Total 

Uneq. 

Eq. 

29D 

0 

1 

1 

29R 

10 

10 

20 

94D 

25 

3 

28 

94R 

3 

1 

4 

Total 

38 

15 

53 


QC commands for unequipped and UC commands for equipped aircraft. 
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Internal clearance coordination by controllers. Sector controllers used the CC function equally for internal 
and external coordination (Tables 6 and 7). Most internal CCs (50 of 54) were trajectory clearances prepared by the 
D-side and sent to the R-side for voice clearance delivery to an unequipped aircraft (24 CCs from 29D to 29R; 26 
CCs from 94D to 94R). 

External clearance coordination by controllers. Most external CCs were used by test sector controllers for 
coordinating conflict resolutions. Of the 53 external CCs, 45 were clearances for aircraft that were in conflict with 
another aircraft, 7 had flight paths bound for weather penetration, and the last was sent by 94R to stop a climb 
from the sector below in anticipation of a complexity increase. 

Unlike internal CCs, the distribution of external CCs showed some between sector differences. In sector 94, the D- 
side again sent most of the CCs (28 of 32), but sector 29's R-side chose to do most of the external coordination. 

This provides one example of how the two participant R-sides chose to divide sector responsibilities differently 
with their respective D-sides. 

3.3.4 Operator Input Errors 

There were eleven recorded input commands recorded at controller positions that were probably operator input 
errors. Four were CC entries that sent apparently accidental requests from an R-side to the D-side, and seven were 
QC entries for equipped aircraft (from both R and D positions) when a UC was probably the intended action. Ten of 
the eleven commands were entered using the keyboard, and one used a fly-out menu. Since the syntax for each 
keyboard input was identical, ten of the entry errors involved only a one-character substitution - "CC [CID]" 
instead "QC [CID]" or "UC [CID]" for the inadvertent CC request, and "QC [CID]" instead of "UC [CID]" for the 
clearance entry error. Note that the CC and QC commands only exist in our simulation environment. 

R-side to D-side CCs. The four internal CCs sent from the R-side to his D-side (1 CC from 29R to 29D; 3 CCs from 
94R to 94D). Camtasia video recordings were reviewed in each case since the reason for these CCs was unclear. 

The Camtasia recordings suggest these four R-to-D CCs appear to have been operator input errors. Each of those 
CCs were followed by a clearance entry (UC or QC) from the same sender within a minute after they were sent, 
and the D-sides were not involved with those clearances. This indicates that the R-side intended to enter a UC or 
QC to complete the clearance, but mistakenly "CC'd" it to the D-side instead. 

QC input errors. As mentioned above, a controller's "QC" input command amends the ATC computer system's 
representation of an aircraft's flight clearance. QCs are normally only used for unequipped aircraft, since the UC 
(Uplink Clearance) command amends the system at the same time it sends the clearance to the equipped aircraft. 
Thus "QC" entries for equipped aircraft may indicate a controller error. 

Each QC event that was logged for an equipped aircraft was reviewed, and 7 of them probably represent controller 
input errors. Table 8 shows the distribution of these errors by run and equipage level, suggesting that equipage 
level may have been a contributing factor: six of the seven occurred in 50% equipage runs, when controllers would 
find themselves switching most frequently between aircraft types, and the last one occurred during a 10% run, 
when most clearances were being developed for unequipped aircraft (Table 9). 

As Table 8 also illustrates, four of the seven events were recorded from the sector 94 radar controller's station. 
Table 9 shows that this controller worked almost exclusively on the unequipped aircraft during the 50% and 10% 
runs, indicating that his D-Side was responsible for sending clearances to the equipped aircraft. It suggests that 
when he had decided to work on a equipped aircraft in runs 4 and 5, he inadvertently used the 'wrong' type of 
clearance entry command. 
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Table 8. Run and Position of Probable QC Input Errors 


Run 

%Equipped 

QC Errors 

Entry Position 

4 

50% 

5 

94R(x3), 98 R, 98D 

5 

50% 

1 

94R 

6 

10% 

1 

29D 


Table 9. UC/QC events, 94R 


Run 

%Equipped 

uc 

QC 

Total 

1 

90% 

19 

7 

26 

8 

90% 

16 

2 

18 

2 

50% 

1 

24 

25 

4 

50% 

0 

23 

23 

5 

50% 

5 

22 

27 

7 

50% 

6 

17 

23 

3 

10% 

0 

19 

19 

6 

10% 

0 

17 

17 

TOTAL 

47 

131 

178 


3.4 Safety Analysis 

Safety metrics included counts of separation violations, conflicts, and convective weather penetration events that 
occurred in one of the four ZKC test sectors. An analysis of separation violations did not uncover any relationship 
to either multi-sector planning or mixed equipage operations. More conflicts were observed in the 50% equipped 
runs, but "open loop" evidence within the 50% scenarios suggests that this was due to the aircraft trajectories and 
not to operator performance at this equipage level. Analyses of weather penetration results for the 50% equipped 
runs found no difference between the numbers of equipped and unequipped aircraft penetrating weather. These 
results indicate that simulation procedures were not inherently riskier for unequipped or equipped aircraft. 

3.4.1 Number of Conflicts 

The number of test area conflicts with predicted separation violations were recorded and analyzed by equipage 
level. Figure 23 presents the mean number of conflicts recorded per run, sorted by equipage level. This figure 
shows that the 50% equipped condition resulted in the greatest number of conflicts followed by the 90%, then 
10% conditions. A one-way ANOVA confirmed that there was a significant difference (F(2, 5) = 7.23, p< .05) in the 
number of conflicts between conditions. Tukey's HSD tests revealed that the 50% condition had a significantly 
higher number of conflicts than the 10% condition (p< .05) but not significantly higher than the 90% condition (p> 
.05). Open loop results presented in the next section suggest that the 50% traffic scenarios were inherently more 
difficult, and that this was the most likely reason for the observed difference in number of observed conflicts. 
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Figure 23. Mean number of conflicts per run 
3.4.2 “Open Loop” Results 

Figure 24 presents a count of separation violations that were logged during the open loop runs. These results show 
that the 50% equipage runs (Traffic 2 scenarios) had more violations than either the 10% or 90% runs (Traffic 1 
scenarios). A breakdown by sector found more open loop violations in sectors ZKC 90 and 98. These results 
indicate that the scenarios designed for each equipage level and each sector were inherently different, with the 
50% scenarios and sectors 90 and 98 being inherently more difficult in terms of aircraft interactions. This between- 
scenario difference is the most likely contributor to the increased number of conflicts and separation violations 
observed during the 50% runs in the simulation. 
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Figure 24. Total number of separation violations recorded in "open loop" runs. 

3.4.3 Separation Violations 

There were a total of 20 separation violations throughout the course of the study. Two were "proximity events" 

(PE) where an aircraft pair had a closest point of approach (CPA) of less than 800 ft vertically and between 4.5 and 
5.0 nm laterally. The rest were "operational errors" (OE) in which the CPA was both less than 800 ft vertically and 
4.5 nm laterally. Figure 25 shows the frequency of these separation violations by equipage level, color-coded by 
the equipage type of the aircraft involved. Sixteen out of twenty separation violations (80%) involved an 
unequipped aircraft; however, analysis of these occurrences did not identify any underlying causal factor related to 
equipage type. 
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Figure 25. Separation violations per equipage level; colors indicate equipage of aircraft pair 


Examination of each separation violation also found that 17 of the 20 involved an aircraft in transition: either a 
departure climbing into the sector, an arrival descending through the sector, or an aircraft given an altitude 
clearance to resolve a separate conflict. This speaks to the difficulties of monitoring the separation status of 
transitioning aircraft, and is unrelated to the MSP concept or its associated functions. 

Finally, Table 10 shows that test sectors 90 and 98 were involved in 17 of the 20 (85%) separation violations. As 
mentioned earlier, these sectors had more conflicts than sectors 29 and 94 during the "open loop" runs; they also 
had more transitioning aircraft flying to and from local airports. In addition, they were staffed by retired 
controllers (rather than FAA test participants) who were possibly not as practiced at controlling traffic, particularly 
at the traffic loads experienced in this simulation. 

Table 10. Total counts and percentages of separation violations per sector 


ZKC 29 

ZKC 90 

ZKC 94 

ZKC 98 

2 (10%) 

7 (35%) 

1 (5%) 

10 (50%) 


3 . 4.4 Weather Penetration 

Three color-coded levels of convective weather intensity were shown on the controller's display, and controllers 
were asked to route aircraft around all intensity levels. Since the weather could shift in unexpected ways, however, 
penetration events were analyzed only for medium and high intensity areas, with the surrounding low intensity 
weather treated as a buffer zone. This approach provided a useful measure of controller effectiveness, and 
excluded "surprise" penetration events that might have been caused by unexpected (or unrealistic) weather 
movement. 

The primary focus of the analysis was on whether there was a difference in the patterns of weather penetration 
between unequipped and equipped aircraft. The 50% equipped condition was used for this comparison. Table 11 
compares the unique number of unequipped and equipped aircraft that penetrated medium or high intensity 
weather in both the simulation and the "open loop" runs. No difference in weather penetration frequency was 
observed between the two equipage type. 


MSP3_Report_3-2-ll 


3/2/2011 


36 



Table 11. Number of unique weather penetration events, by equipage type, in simulation and "open loop" runs. 


Simulation 


"Open Loop" 

Unequipped 

Equipped 


Unequipped 

Equipped 

12 

14 


5! 

53 


3.5 Subjective Feedback 

In this section, we first examine in-run workload ratings reported by the four radar controllers (two participants 
and two confederates), the four radar associates, the two area supervisors, and the three ZKC TMU positions (two 
participants and one confederate). In the remainder of the section, we report and summarize subjective feedback 
collected in post-run and post-simulation questionnaires and a post-simulation debrief discussion from the six test 
participants only, except where otherwise indicated. 

3.5.1 Workload 

Instantaneous workload ratings, on a scale from 1 (lowest) to 6 (highest), were obtained from all test positions: the 
three ZKC TMU positions, two area supervisors, four radar controllers, and four radar associates. Ratings were 
recorded every five minutes throughout each run. 

As expected, radar controller and radar associate mean workload decreased as equipage level increased (Figure 
26). A repeated measures ANOVA confirmed a significant difference between conditions (F(2,14) = 12.91, p < .01) 
and follow-up planned contrasts revealed mean workload was significantly lower in the 50% condition than in the 
10% condition, and significantly lower in the 90% condition than in the 50% condition (Fs(l,7) = 6.04 and 17.16, ps 
< .05 and .01, respectively). 

For area supervisors and the three ZKC TMU participants, mean workload was lower in the 90% condition, but did 
not differ between the 10 and 50% conditions (Figure 26). A repeated measures ANOVA revealed a significant 
difference between conditions (F( 2,8) = 21.84, p < .01) and follow-up planned contrasts revealed workload was 
significantly lower in the 90% condition than in the 50% condition, but did not differ between the 50 and 10% 
conditions (Fs(l,4) = 31.36 and 0.02, ps < .01 and >.05, respectively). 



Figure 26. Mean workload rating per position and equipage level. 
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3.52 Task Load (NASA-TLX) 

After each run, each of the six participants responded to a modified version of the NASA-Task Load Index 26 (NASA- 
TLX), rating each of the six factors, (performance success, effort, frustration, and mental, physical, and temporal 
demand) based on their peak workload during the run (Appendix C). Each factor was rated on a seven point scale 
ranging from very low / very little to very high / a lot. An average rating, using the scale inverse of the Success 
rating (Rating inverse = (-l)Rating origina | + 8) was computed for each position x equipage level combination. 

In general, task load ratings decreased, and perceived performance success increased, as equipage level increased 
(Table 12). The ratings for the 10 and 50% conditions also tended to be more similar to each other than to those 
for the 90% condition. This observation was confirmed statistically: A repeated measures ANOVA on average TLX 
score, with within subjects variable Equipage Level, was significant (F(2,10) = 5.27, p < .03) and post hoc contrasts 
revealed significant and marginally significant differences between the 50 and 90% conditions and between the 10 
and 90% conditions, respectively (ts(5) = 3.68, 2.20, ps < .05, .10, respectively), but no difference between the 10 
and 50% conditions (t(5) = 0.17, p > .05). 

One TLX subfactor, Frustration ("Were you frustrated by this run ? e.g., were you discouraged, irritated, stressed, 
and annoyed, or were you content, relaxed, gratified, and complacent when performing the task?"), was 
consistently higher in the 50% than in the 10 or 90% conditions, across positions. That is, controllers, area 
supervisors, and STMC/TMCs all found the 50% condition the most frustrating. This observation was confirmed by 
paired-samples T tests, which revealed significant and marginally significant differences between the 50 and 90% 
conditions and between the 10 and 50% conditions, respectively (ts(5) = 3.21, -2.45, ps < .05, .10, respectively), but 
no difference between the 10 and 90% conditions (t(5) = 0.17, p > .05). 

Comparing across positions, radar controllers tended to give higher task load ratings than area supervisors and the 
STMC / TMCs, but only in the 10 and 50% conditions. In the 90% condition, all positions reported a low (~2.5 out of 
7) average task load. 

Task load vs. workload. The post-run NASA TLX ratings were compared with the in-run workload ratings that 
participants entered every 5 minutes. For the two participant controllers, all 6 TLX factors were significantly 
correlated with both the mean and the maximum WAK rating per run, with Effort correlating most strongly with 
mean WAK (Pearson's r = .90, p < .001), and Physical Demand with max WAK (r = .87, p < .001). For the two area 
supervisors, all 6 factors were correlated with mean WAK rating per run, though Success was only marginally 
significantly correlated (p < .10). Just as for the controllers, area supervisor Effort was the factor most strongly 
correlated with mean WAK (r = .79, p < .001). For area supervisors however, no TLX factors were correlated with 
the maximum WAK per run at the p < .05 level. For the two TMU participants, no TLX factor was correlated at the p 
< .05 level with either mean or max WAK. 
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Table 12. Mean NASA-TLX Subscale Ratings 


Mean rating for each of the six subscales of the NASA-TLX, and the average* of the six subscales per condition. 
Ratings made on a seven point scale ranging from 1 = very low / very little to 4 = average to 7 = very high / a lot. 


Equipage Level 


10% 

50% 

90% 

Radar Controllers Mental Demand 

5.50 

4.88 

3.25 

Physical Demand 

5.50 

4.50 

3.00 

Success 

5.50 

4.63 

6.25 

Effort 

5.25 

4.63 

2.75 

Temporal Demand 

4.50 

3.50 

2.25 

Frustration 

3.25 

3.50 

1.75 

Average 

4.42 

4.06 

2.46 

Area Supervisors Mental Demand 

3.25 

3.75 

3.50 

Physical Demand 

2.50 

2.75 

2.00 

Success 

4.25 

4.50 

5.50 

Effort 

3.00 

3.38 

2.75 

Temporal Demand 

2.00 

3.00 

2.25 

Frustration 

2.00 

2.38 

2.00 

Average 

2.75 

3.13 

2.50 

STMC/TMC Mental Demand 

3.50 

3.50 

2.75 

Physical Demand 

4.00 

2.75 

3.00 

Success 

5.25 

5.13 

6.50 

Effort 

4.50 

3.50 

3.25 

Temporal Demand 

2.75 

2.75 

2.50 

Frustration 

2.00 

2.25 

1.50 

Average 

3.25 

2.94 

2.42 


3.5.3 Acceptability 

After each run, each of the six test participants responded to a modified version of the Controller Acceptance 
Rating Scale (CARS) which was presented by computer. 27 The acceptability rating scales used for each position - 
radar controller, area supervisor, and STMC/TMC - are shown in flowchart format in Appendix D, although they 
were presented to participants by computer as a progressive set of questions and statements. After the initial 
question in each scale, subsequent questions/statements were presented conditional upon the response to the 
previous question. At each decision point, participants were allowed to change their response and to review all the 
possible responses before making their final selection. 

In general, the acceptability rating increased with the equipage level (Figure 27). Figure 27 shows the ratings given 
across positions and equipage levels, grouped according to the decision points on the modified CARS: Is it 
safe/workable? (No — > 1); Is workload tolerable? (No — > 2-4); Are operations satisfactory without improvement? 
(No — » 5-7); Flow desirable? (8-10). Each dot represents one participant's rating on one run and the position of the 
dot within the box aligns with the response (e.g., a rating of "10" appears in the rightmost position in the 8-10 
box). In all cases the modified CARS description and rating selected were thought to adequately match the 
perceived acceptability of the operations (i.e., no one indicated that the description and rating matched their 
experience "not well"). 


* The success rating scale was reversed (e.g., 1 = 7, 1.5 = 6.5, etc.) when calculating the six-subscale averages. 
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As can be seen, radar controller ratings clustered in the 8-10 range across equipage levels, indicating that the radar 
controller role was satisfactory at all three equipage levels tested. There were two cases, one each in the 10 and 
50% equipped conditions, where a controller rated the position as workable but with excessive workload (i.e., in 
the 2-4 range). In both cases, however, the low rating appeared to be due to factors unrelated to the mixed 
equipage and service for equipage operational concepts (e.g., complexity due to weather and traffic volume). In 
each case the controllers responded that the equipage level itself was not problematic or confusing. 


The area supervisor ratings generally clustered in the 8-10 range across equipage levels, indicating that operations 
were satisfactory at all three levels tested. There were three cases in the 50% condition and one in the 10% 
condition of ratings in the 5-7 range, suggesting operations were safe and workload was manageable but that 
improvement was needed. All four of these ratings came from the ZKC North area supervisor, who indicated that 
traffic volume and in particular the unequipped aircraft added to the workload. 


The STMC / TMC ratings clustered in the 8-10 range across equipage levels, indicating that operations were 
satisfactory at all three levels tested. There was one low rating, on a 10% run, where the participant acting as 
STMC indicated that operations were safe but workload was too high because the volume of traffic that needed to 
be rerouted placed too great a demand on his team for the first twenty minutes of the run. Again, this appeared to 
be the result of the initial traffic scenario design, rather than related to the operational concepts being tested. 
Interestingly, the other TMC test participant rated that same run as a "9". 
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50% Equipped 


90% Equipped 


Radar Controllers 

1 2-4 5-7 8-10 

D Q D E 


1 2-4 5-7 8-10 



1 2-4 5-7 8-10 



► 

Acceptability 



Area Supervisors 


2-4 5-7 8-10 


2-4 5-7 

~rr 

8-10 

LJUl 

h 




► 

Acceptability 



STMC / TMC 




► 

Acceptability 


Figure 27. Acceptability ratings by position and equipage level. 


3.5.4 Equipage Levels 

Several post-run questions explicitly asked test subjects about their experiences with respect to the equipage level 
of the just-completed run, and other questions and post-run ratings were analyzed to determine the possible 
impact of equipage level. Subjective results related to equipage level and its impact are presented in this section. 

Controllers. After each run, participant controllers were asked whether the equipage level ever made things 
confusing. There were only two cases where controllers responded "yes," and both were for 50% runs. In the first, 
the controller indicated that the confusion was due to difficulty dividing roles between radar controller and radar 
associate. In the second, the difficulty cited was knowing where the unequipped were landing, and starting them 
down both in a timely manner and free from conflicts with the equipped, which descended on their own. 
Controllers did, however, indicate that the equipage levels and traffic loads were not problematic. About half the 
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time, controllers indicated that they could have handled more traffic in the 10 and 50% runs, and almost always in 
the 90% equipage runs. 

On scales from 1 ("not at all") to 4 ("moderate") to 7 ("very"), mean controller ratings of safety were moderate 
(about 5) for 10 and 50% runs, and very safe (6.75) for 90% runs. Mean traffic complexity ratings decreased with 
increasing equipage levels, from moderate for 10 and 50% runs (Ms = 5.25 and 4.25, respectively) to low-to- 
moderate for 90% runs (M = 2.75). Rated effectiveness in managing unequipped aircraft was higher in 90% runs (M 
= 6.33) than in 10 or 50% runs (Ms = 4.75 and 4.88, respectively), but effectiveness in managing equipped aircraft 
was high (Ms > 6.2) across equipage levels. Rated difficulty of keeping unequipped aircraft on trajectory was 
moderate for 10 and 50% runs (Ms = 3.75 and 4.13, respectively) and low for 90% runs (M = 2.25). 

Area Supervisors. Area supervisor of ZKC South thought the traffic and equipage levels were manageable on 
every run; area supervisor of ZKC North found them barely manageable on two runs, one 10% and one 50%, but 
manageable on all other runs. In both cases, area supervisor for ZKC North thought the equipage levels, 10 and 
50%, would have been manageable under lower traffic levels. Area supervisors thought controllers could have 
handled higher traffic levels on about half of the 10 and 50% runs and about 75% of the 90% runs. 

Traffic management coordinators. One TMU participant rated the equipage level and traffic load as 
unmanageable on one 10% run, but thought it would have been manageable with more TMCs on staff. The 
participant further explained that the manageability increased over the course of the run: "Work was very 
intensive for the first 30 minutes of the run. After getting a handle on the traffic, then simple maintenance was all 
that was necessary to keep the sectors under control." This is likely the result of our traffic scenario design, which 
progressively increased traffic to a relatively high level - particularly for a 10% equipped environment, over 
roughly the first third each run. On every other run, both TMU test participants considered the equipage levels and 
traffic loads to be manageable as staffed. TMU participants thought they could have managed higher traffic loads 
on 50 and 90% runs. Both TMU participants thought that they may have been able to manage a higher traffic load 
on run 8, the third 10% run. This suggests a relatively quick learning effect that increased the acceptability and 
manageability of the simulation traffic loads in the 10% equipped condition. 

3.5.5 Service for Equipage 

Area supervisors and traffic management coordinators. Across equipage levels, area supervisors and 
traffic management coordinators reported that, whether moving traffic for sector load or weather, they were able 
to provide better service to the equipped aircraft relative to the unequipped aircraft, by rerouting unequipped 
aircraft first and allowing equipped to fly their original, or closer to their original, trajectory. For weather, that 
meant unequipped aircraft were rerouted and equipped aircraft were allowed to just "skirt" weather. One 
supervisor commented that the service for equipage policy was a "win-win," since moving an unequipped aircraft 
out of a sector lightens that sector controller's workload more than moving an equipped aircraft, and at the same 
time rewards equipped aircraft with better service. Traffic management coordinators indicated that a strategy of 
focusing on the unequipped aircraft, and leaving the equipped untouched, helped them to resist the inclination to 
move the equipped first simply because it was operationally easier to do so. When feasible, the TMC left it to the 
controller to move an equipped aircraft, if necessary. 

TMCs also pointed out that the varying equipage levels did not change their general strategy for providing better 
service to equipped aircraft, but did affect how well they could provide better service. In 10% equipped runs, 
equipped aircraft were simply "not touched" by the TMU. In 90% runs, they felt they had no choice but to reroute 
some equipped in addition to the unequipped aircraft. TMCs also mentioned that if they noticed a sector "going 


MSP3_Report_3-2-ll 


3/2/2011 


41 


red," but that it was equipped aircraft sending it over the limit, they would "let it ride," since the equipped aircraft 
were "not a big deal" for controllers. 

Controllers. Across equipage levels, controllers responded that they gave equipped aircraft priority access to 
constrained airspace and made unequipped aircraft yield to equipped aircraft in mixed equipage conflicts. In both 
cases, the mean difficulty rated on a 1 ("not at all") to 7 ("very difficult") scale was 2 or lower for all three equipage 
levels. 


3.5.6 Team Configuration, Roles, and Responsibilities 

Area supervisors. One area supervisor participant, ZKC South, made use of the multi-sector planning station 
while the other, ZKC North, did not. The area supervisor for ZKC South preferred to work from the planning station 
in the corner of the room because the additional, larger screens made the tools easier to use. The area supervisor 
for ZKC North was concerned that sitting at the planning station, rather than at his smaller but more centrally 
located station, would remove him from the operation and cause him to lose situational awareness. This concern 
appeared justified since there was one observed instance when the South supervisor, sitting at the planning 
station, was too late in noticing that his sector 29 radar controller needed help. 

Both area supervisors felt that their roles and responsibilities in the simulation did not differ substantially from their 
actual roles in the facility today. They did point out that they were able to move aircraft more dynamically and 
closer-in than they can today, and were less reliant on the TMU since they could coordinate with each other and 
reroute aircraft directly. They also pointed out that their workload and sector complexity decreased as equipage 
level increased, but that during 90% equipped runs, they could no longer rely on "voice (speed, volume, or 
inflection) to indicate how busy a controller was." The mean difficulty of determining when to assign/remove radar 
associates, rated on a 1 ("very easy") to 5 ("very difficult") scale, was higher for 50 and 90% runs (1.88 and 1.75, 
respectively) than for 10% runs (1.00). 

ZKC South supervisor indicated that across equipage levels he spent about 50% of his time monitoring the 
controllers and potential problems in individual sectors, 30% developing solutions for problems in his area, 10% 
executing TMU's plans, and 10% requesting assistance from the TMU. ZKC North supervisor did not estimate task 
distribution by percentage of time, but noted that the more unequipped aircraft there were, the more time he 
spent monitoring the radar controllers. He also pointed out that problems in his area were easier to resolve with 
equipped aircraft. 

Traffic management coordinators. The participant acting as STMC in each run took charge of within and 
between center coordination and determined the division of responsibilities between his TMCs. Participants 
indicated that they tried three main divisions of responsibility between the two TMCs working the ZKC test area: 
geographic (North / South Center split); weather / volume (1 TMC reroutes for Wx / 1 TMC manages sector volume 
& complexity); and altitude (FL290 - 350 / FL360 - 600). Both believed the division by altitude strata led to the 
fewest cases of overlap or duplication of effort between TMCs. They also explained that the effectiveness of the 
division of roles largely depended upon the equipage level and weather in the run, and that they sometimes 
needed to adjust the assigned roles in the middle of a run in order to accommodate changes in the situation. The 
importance of communication between all parties and STMC oversight for reducing overlap was stressed. 

Comparison with 20 og multi-sector planner simulation. Our two TMU participants, both of whom had 
previously participated in the 2009 Multi-Sector Planner Simulation, were asked to compare the effectiveness of 
operations and team configuration in the previous simulation with the current simulation. Both responded that 
operations worked better in the current simulation and that they felt they were better able to accomplish their 
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objectives with less duplication of effort. They attributed their improved effectiveness and efficiency to clearer 
delineation of responsibilities and oversight of the MSP function by the STMC, and cited a lack of organization and 
supervision of MSP activities in the 2009 study. 

The two area supervisor participants in the current study were asked whether they could have used a multi-sector 
planner TMC located in the area ("on the floor"). Both believed the multi-sector planning functions would be 
useful but that a supervisor could handle the responsibilities, with no need for a TMC to staff the position. 

Radar controller vs. radar associate Radar associate positions were staffed on test sectors 29 and 94 for at 
least part of every run, and for the entire run for about 80% of the runs. Having the same scope and tools, radar 
controllers and radar associates each had the capability to perform any ATC task, though only radar controllers 
spoke to pilots. In post-run questionnaires, various ATC activities were listed and the radar associates for all four 
test sectors indicated which of those they had performed on the previous run. Table 13 summarizes their 
responses by equipage level. Radar associates acted as a second set of eyes and ears, helped with handoffs, and 
looked for conflicts across all three equipage levels. As equipage levels increased, radar associates resolved fewer 
mixed equipage conflicts and conflicts between unequipped aircraft, and more conflicts between equipped 
aircraft, presumably because the number of unequipped aircraft in their airspace decreased. They also spent less 
time rerouting aircraft around weather and putting aircraft back on trajectory ("on track"), as equipage level 
increased. 

Radar controllers and associates were asked whether there was any confusion, unnecessary overlap, or duplication 
of effort within sector teams. Of the fifteen instances they mentioned, eleven occurred (73%) during 50% 
equipped runs, suggesting that radar controller - radar associate role division was trickier in 50% scenarios. 

Table 13. Mean percentage of runs on which radar associates reported performing particular ATC tasks. 



Equipage Leve 

1 

ATC Task 

10% 

50% 

90% 

Second set of eyes and ears 

100.0 

93.8 

100.0 

Accept & initiate handoffs 

100.0 

93.8 

100.0 

Look for conflicts 

100.0 

87.5 

100.0 

Resolve conflicts between EQUIPPED A/C 

50.0 

62.5 

62.5 

Resolve conflicts between UNEQUIPPED A/C 

50.0 

43.8 

25.0 

Resolve MIXED CONFLICTS (i.e., conflicts involving 
equipped and unequipped A/C) 

50.0 

50.0 

25.0 

Reroute A/C around weather 

100.0 

100.0 

87.5 

Put free-track A/C back on route 

100.0 

87.5 

75.0 

Coordinate with other controllers (e.g., on handoffs, 
A/C not on frequency, conflict resolution) 

100.0 

87.5 

87.5 

Data block management or clean-up 

75.0 

68.8 

75.0 


Radar associates were also asked in post-run questionnaires how they performed certain air traffic control tasks, 
and selected from a list of options (Table 14). Specifically, radar associates were asked how they trial planned and 
coordinated reroutes for unequipped, equipped, and un-owned (regardless of equipage) aircraft. In managing 
unequipped aircraft, radar associates trial planned routes and coordinated those routes with their sector radar 
controller using data exchange (i.e., "CC") much more often than verbal communication. In managing equipped 
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aircraft, radar associates most often uplinked routes directly to aircraft, rather than sending them to the radar 
controller's station for review and uplink. When it was necessary to manage un-owned aircraft (e.g., for a mixed 
equipage conflict in which the unequipped aircraft was un-owned), radar associates most often coordinated 
verbally with the other sector controllers, rather than via ground-ground data coordination. 

In the post-simulation questionnaire, the two participant radar controllers (for sectors 29 and 94) were asked to 
reflect on their experiences across equipage levels and give their opinion on the optimal division of tasks between 
radar controller and radar associate. Describing the optimal division, both split certain duties according to whether 
the aircraft were equipped. Both allowed their radar associates to send clearances for equipped aircraft directly to 
the aircraft (via a "UC"), but required clearances for unequipped aircraft be sent to them (via a "CC"). Both 
described the radar associate ability to trial plan reroutes and send ("CC") these for review and issuance as very 
useful. 

More specifically, the sector 29 controller thought the radar controller should be more focused on the unequipped 
aircraft and the radar associate on the equipped aircraft, but with the radar controller taking all handoffs. The 
radar associate would assist in initiation of handoffs. The radar controller and associate would split the duties of 
rerouting aircraft around weather and around conflicts when able. 

According to the sector 94 controller, the radar controller would look at and resolve all conflicts, initiate/accept 
handoffs on unequipped aircraft, trial plan descents/climbs, manage the data blocks, and respond to coordinated 
clearances ("CCs"). The radar associate would look at and resolve weather reroutes for equipped aircraft, look at 
and coordinate (via a "CC" to the radar controller) weather reroutes for unequipped aircraft, assist in initiating 
handoffs on unequipped, and assist in data block management. 

Both sector 29 and 94 controllers stated that the division of tasks would change as the equipage level changed. 
With fewer aircraft equipped, the radar associate would have to assume more duties because the radar controller 
was so busy issuing voice clearances. With 90% equipped, the radar associate might only assist with data block 
management, depending of course on the actual level of complexity due to, for example, weather or conflicts. 


Table 14. Mean percentage of runs on which radar associates reported performing ATC tasks in particular ways. 



E 

quipage Level 


ATC Task 

10% 

50% 

90% 

Trial plan UNEQUIPPED & send ("CC") to R-Side to issue verbally 

87.5 

75.0 

25.0 

Trial plan UNEQUIPPED and VERBALLY COORDINATE (no "CC") the 
new route with R-SIDE to issue verbally to A/C 

37.5 

31.3 

0.0 

Trial plan EQUIPPED & send ("CC") to R-Side to uplink 

25.0 

31.3 

25.0 

Trial plan EQUIPPED & UPLINK directly ("UC") to A/C 

100.0 

100.0 

87.5 

Trial plan UN-OWNED A/C & send ("CC") to OTHER CONTROLLERS 

50.0 

50.0 

25.0 

Trial plan UN-OWNED A/C & VERBALLY COORDINATE with OTHER 
CONTROLLERS 

100.0 

100.0 

100.0 


3.6 Tools Feedback 

In this section we summarize subjective feedback from the post-simulation tools questionnaire from the six test 
participants only. The tools questionnaire covered, among other things, the usefulness and usability of the multi- 
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sector planning and controller tools in a mixed equipage environment. Most questions were asked for both 
equipped and unequipped aircraft. Only feedback and ratings which bear directly on the experimental questions of 
the tools' adequacy in a mixed equipage environment are summarized here. Appendices C and D provide more 
complete and detailed ratings data on all the tools. 

3.6.1 Radar Controller Tools 

Radar controllers and radar associates had use of the full DSR tool set. This tool set included: Route and Altitude 
Trial Planning (for both equipped and unequipped aircraft), Conflict Detection, Weather Probing, Automatic 
Trajectory Resolutions for conflict or weather avoidance, Data Comm equipped aircraft options and unequipped 
aircraft voice communication options, as well as ground-to-ground Data Comm. 

Controller tools in a mixed equipage environment. Tool usefulness vs. usability questions were broken 
down by equipage type. For each set of radar tools, the controllers were asked how useful or usable they were for 
both equipped (E) and unequipped (UE) aircraft. Figure 28 and Figure 29 below present the mean usefulness and 
usability ratings by the two radar controller participants, only. Figure 28 shows the radar controller tool ratings for 
equipped and unequipped aircraft. Figure 29 presents the usefulness and usability ratings of the communication 
tools, Data Comm and voice communication, for equipped and unequipped aircraft. More detailed ratings data on 
all radar controller tools can be found in Appendix E. 


R-Side Tools: Usefulness v. Usability for Equipped and Un-equipped Aircraft 
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Figure 28. R-Side Tools: Usefulness v. Usability for Equipped and Unequipped Aircraft 
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R-Side Communication Tools: 

Data Link Equipped v. Voice Un-Equipped Usefulness and Usability 



I Equipped-Data Link 
Un-Equipped-Voice 
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Not Useful/Usable (1) to Very Useful/Usable (6) 


Figure 29. Communication Tools: Data Comm Equipped v. Voice Unequipped Usefulness and Usability 


In general, feedback on the controller tool set was positive. Controllers found the trial planning tools highly useful 
and usable with both equipped and unequipped aircraft. One issue cited, however, was that the lack of all the 
ground based NAVAIDS made it difficult to trial plan a good reroute for unequipped aircraft. Also, the direct TR 
keyboard reroute function was rated relatively low in usefulness; the two participant controllers did not use this 
function because they were not familiar enough with the test airspace and because they really liked using the click- 
and-drag trial planning function. Controllers loved the conflict probe feature for both equipped and unequipped 
aircraft and one controller commented, "Amazing, get this on our scopes now!" They also liked the weather 
penetration prediction tool but thought the weather avoidance reroute feature needed a bit more "forecasted" 
information. 
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Working in a mixed equipage environment meant that each radar controller could control more traffic than in 
normal, present-day operations: Data Comm enables automated transfer of communication, eliminates the voice 
check-in requirement, and improves performance of trajectory-based tools. Because task load is reduced with 
equipped aircraft, aircraft count increases at higher equipage levels. The more aircraft per sector, the more the 
data blocks overlapped visually. One controller indicated that he would have liked a FDB "de-conflict button," to 
spatially separate overlapping data blocks. Further, equipped aircraft were not required to check-in as they 
entered a new sector. Consequently, controllers also indicated that making it easier to locate specific aircraft on 
their scopes would help with situational awareness. One suggestion was an alphabetical list of aircraft to select 
from that, when selected, would highlight the corresponding FDB on the scope. 

Working in a mixed equipage environment required controllers to manage and interact with aircraft differently 
depending on the equipage type. For example, controllers had to manage two different types of transfer of 
communication, automatic and manual. Automatic transfer of communication for equipped aircraft was rated high 
in both usefulness and usability; one controller said, "Get this to us now." Usefulness and usability of manual 
transfer of communication was rated lower, and one controller commented that the usability dropped as the 
number of unequipped aircraft increased. In contrast, the mechanism for issuing clearances was well accepted for 
both equipped (UC) and unequipped (QC) aircraft, but controllers thought that Data Comm (UC), in particular, 
greatly reduced their workload, contributing to higher safety and their ability to handle more aircraft. With 
unequipped aircraft, controllers indicated that they had to use more "props," like the J-ring, to remind them when 
they needed to do something. Overall, however, controller feedback suggested that the available tools made it 
possible to work the traffic in a mixed equipage environment. 

Synchronized radar controller and radar associate scopes. Another prototyped feature of the controller 
team tool set was the synchronization of the radar controller and radar associate scopes. The participants were 
asked how useful or usable the synchronized display concept was in terms of various controller tools. Figure 30 
presents the two R-Side participant responses to the synchronized display concept. In general, controllers found 
the R-Side-D-Side scope synchronization acceptable. One participant commented that there were times when he 
would have liked to off-set the data blocks differently on his scope than on his D-Side's scope, but he also 
remarked that in a high workload environment it was probably better to have the data block locations the same. 


R-side and D-side Display Synchronization: Usefulness v. Usabilty 
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Figure 30. R-Side and D-Side Display Synchronization: Usefulness v. Usability 
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3.6.2 Multi-Sector Planning Tools: STMC/TMC and Area Supervisors 

The two TMU participants and two Area Supervisors had use of the full multi-sector planning tool set. This tool set 
included: Load Display Control Windows, Load Table, Load Graphs, DSR, TSD (weather information), AC Filters 
tools, Traffic Monitoring and Problem identification. Solution Planning and Solution Coordination/Communication. 
Only feedback which bears directly on the experimental questions of the tools' adequacy in a mixed equipage 
environment is examined here. 

Multi-sector planning tools for mixed equipage. Figure 31, Figure 32, and Figure 33 present the mean 
usefulness and usability ratings by the area supervisor and TMU participants. The tools ratings are presented in 
three figures organized into the following three categories: MSP Tool Sets (General), Trial Planning Tools, and 
Communication Tools. Figure 31 presents the overall usefulness and usability ratings of the major MSP tools as 
they were used for two primary functions: 1) traffic monitoring and problem identification; and 2) actual solution 
planning. Figure 32 shows the usefulness and usability results for the trial planning tools. Figure 33 shows the 
usefulness and usability of both the voice and Data Comm tool set. More detailed ratings data on all MSP tools can 
be found in Appendix F. 
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Figure 31. Overall Usefulness and Usability of MSP Tool Set 
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Ovearll Usefulness and Usability of Trial Planning Tools 
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Figure 32. Overall Usefulness and Usability of Trial Planning Tools 

MSP tools and service for equipage. In their comments, area supervisor and TMU participants indicated that 
the filter tool made it easier to provide service for equipage. By filtering out the equipped aircraft and displaying 
only the unequipped aircraft, participants were able to focus their initial attempts at managing sector complexity 
on traffic initiatives that only affected the unequipped aircraft, leaving the equipped to remain on their original 
trajectories. In rerouting unequipped aircraft, however, participants felt that the waypoints available for trial 
planning were too sparsely spaced. They would have liked more closely-spaced waypoints to choose from, in order 
to reduce the reroute penalty to unequipped aircraft. Both area supervisors and TMU participants also commented 
that in using the load table and graphs to manage sector load, they allowed higher sector counts when a majority 
of the aircraft was equipped, and TMU participants allowed sector predictions above the nominal Monitor Alert 
Parameter (MAP) of 22 if the aircraft sending the sector over MAP were equipped aircraft. 
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Overall Usefulness and Usability of Communication Tool Set 
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Figure 33. Overall Usefulness and Usability of Communication Tool Set 


Distribution of multi-sector planning tools. Area supervisors indicated they made use of all the tools 
available to them, especially during peak periods, to manage sector complexity. For example, area supervisors 
commented that they used the filters to identify aircraft that required reroutes for sector load or weather 
avoidance and that the tools allowed for easy analysis of future traffic. Doing so, however, was easier from the full 
planning work station in the corner of each control room which, they felt, caused them to have their back to the 
floor for too long; they would have liked the full station to be more centrally located, with a view of controllers. 


With area supervisors using the multi-sector planning tools to reroute traffic, TMU participants were concerned 
about duplication of efforts. One function they felt was missing and would have found useful was a way to see 
whether a given plane had been moved previously, and if so, how many clearances it had been issued. Without 
such a tool, they were concerned that aircraft might get issued multiple conflicting clearances, if not done with an 
organized, high-level plan in mind. 

An important tool for planning and coordination, ground-to-ground data exchange tools were rated high in both 
usefulness and usability by area supervisors and TMU participants. Participants also appreciated ground-to-ground 
data exchange since it reduced the amount of verbal coordination they needed to do. 
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4 Discussion 

A HITL simulation was conducted to determine the feasibility and value of conducting multi-sector planning (MSP) 
operations in a mixed equipage environment. The simulation also provided insight on tool requirements (for both 
planner and controller stations), as well as planning and coordination procedures and distribution of roles and 
responsibilities both within facility (TMU/Operational Area) and within sector (R/D). 

This simulation also allowed us to explore the application of a proposed NextGen "best-equipped, best-served" air 
traffic management policy. Procedures for accomplishing this at the sector and multi-sector level were developed 
and tested, and a rich set of results were obtained that speak to its effectiveness and feasibility, as well feedback 
from participants about its operational suitability. 

4.1 Feasibility 

The operational feasibility assessment addressed two related questions: (1) are MSP operations feasible for 
unequipped aircraft, and (2) are they feasible in a mixed equipage context. The simulation allowed us to explore 
other feasibility topics that were not directly coupled to this objective, including the feasibility of mixed equipage 
operations in general, and of procedures to support trajectory-based operations for non-Data Comm aircraft. 

4.1.1 Operational Feasibility 

Overall, the results indicated that the simulated MSP tools and procedures were effective with both equipped and 
unequipped aircraft, and that these operations were feasible in a mixed equipage environment. Trajectory 
coordination proved feasible for unequipped aircraft as well as for equipped aircraft. Roughly two thirds of all 
coordinated clearance requests were for unequipped aircraft, and nearly all coordinated clearances were accepted 
and executed by the receiver. 

Using the MSP tools, TMCs were able to manage controller task load, effectively balancing airspace throughput 
with sector complexity and controller task load at each equipage level tested. Operator feedback was consistent 
with performance and usage metrics. Across equipage levels and positions, mean reported task and workload 
remained at average and tolerable levels, and CARS operational acceptability ratings were satisfactory. 

4.1.2 General Mixed Equipage Feasibility 

The simulation also shed light on potential feasibility considerations for mixed equipage operations more 
generally. Although no equipage level was considered particularly problematic, the 50% level was more challenging 
than either the 10% or 90% levels. In the post-run TLX questionnaire, controllers, area supervisors, and TMC 
participants all reported the 50% runs to be the most frustrating, although the reported mean frustration level was 
still at or below the middle value for all 3 participant categories (traffic management, supervisors and controllers). 
One controller explained that the 50% runs were hardest because they required controllers to make more service- 
for-equipage decisions. That is, the additional cognitive load placed on controllers by asking them to consider 
providing priority service to the equipped aircraft will be highest when the likelihood of mixed equipage situations 
is highest (i.e., at the 50% level). One suggestion that was offered for mitigating this additional workload was for 
the automation to assist the controller in providing service for equipage by offering resolution advisories that 
preferentially move the unequipped aircraft in mixed equipage conflicts. 

The 50% equipage level was also reported to be the most confusing in terms of dividing roles and responsibilities 
between radar controllers and radar associates. Overlap and duplication of effort - for example, both team 
members working on rerouting the same aircraft - reportedly occurred most frequently during 50% runs. Finally, 


MSP3_Report_3-2-ll 


3/2/2011 


51 


the 50% runs also had the highest number of controller input errors - i.e., using the wrong action to execute a 
clearance for an equipped aircraft. 

4.1.3 Feasibility of TBO for Unequipped Aircraft 

One of the key choices for this study was the decision to plan and coordinate trajectory-based clearances for both 
equipped and unequipped aircraft. This meant that tools and procedures were needed to support trajectory-based 
operations for unequipped planes, so a trial plan mechanism for creating and issuing clearances based on named 
waypoints was developed. The aircraft target symbol was modified, too, to provide salient feedback when aircraft 
were out of conformance with the ATC system-entered trajectory, and one regular D-side task was to update the 
system entry as needed to bring it into conformance with the actual flown trajectory. 

Participants found the tools for trial planning trajectories to be effective for unequipped aircraft. The named 
waypoints available to them for route construction were not as dense as they would have liked, however, and they 
developed workaround techniques for minimizing any consequent route inefficiencies. If, for example, the 
waypoint-based reroute for weather avoidance took an unequipped aircraft farther from its preferred route than 
necessary, they would plan to return the aircraft back on course after the weather was no longer a concern. This 
technique of issuing a second follow-up clearance to avoid excessively inefficient reroutes increased the 
controller's workload, but was used regularly to compensate when the available waypoint's location was not 
optimal. A solution proposed by several participants was to increase the density of named waypoints, perhaps by 
adopting the NRS grid protocol. 29 

4.2 Benefits 

Two categories of MSP-associated benefits were evaluated in this simulation: system performance improvements 
and priority service for equipped aircraft. System benefits were also observed with increasing equipage levels, 
independent of MSP operations. 

4.2.1 Benefits Associated with MSP Operations 

System benefits. System performance improvements were evident both in terms of test airspace throughput 
and controller workload. Using the MSP tools, TMCs selectively diverted aircraft away from the test airspace in 
order to manage predicted sector complexity, a metric that weighted unequipped aircraft more heavily than 
equipped aircraft. As a result, higher throughput was maintained as equipage levels increased, without elevating 
controller workload. In fact, controller workload decreased as equipage levels increased. 

Service for equipage. A strategy of selectively diverting unequipped aircraft also provided priority service to the 
equipped aircraft. TMCs and, to a lesser extent, area supervisors, could selectively permit equipped aircraft 
increased access to congested airspace. The net results were higher test airspace throughput for equipped than 
unequipped aircraft and greater path length increases across all equipage levels for unequipped aircraft. 

4.2.2 System Benefits of Higher Equipage Levels 

With the multi-sector planning operations managing airspace complexity, higher throughput was observed with 
higher equipage levels - i.e., fewer aircraft needed to be routed around constrained resources to keep demand at 
a manageable level. Equipage-related workload differences were so pronounced that increased throughput was 
accompanied by reduced workload at higher equipage levels. This suggests that the complexity calculation could 
have been adjusted to increase the weighting difference between equipage types, allowing even higher 
throughput at the 90% equipage level. 


MSP3_Report_3-2-ll 


3/2/2011 


52 


The increased overall efficiency with higher equipage levels had an interesting consequence in terms of service for 
equipage as well. Even though controllers and planners reported having fewer opportunities to selectively provide 
priority service to equipped aircraft at the higher equipage levels, the observed advantage to equipped aircraft (in 
terms of path length, throughput differences and conflict resolution maneuvers) was maintained, even at the 90% 
equipage level. This was probably because system efficiency overall was improved: the (10% minority of) equipped 
aircraft saw a comparative benefit in the low-equipage case, while the (90% majority of) equipped aircraft saw a 
comparative benefit in the high equipage case. Note that the converse also appeared to be true: unequipped 
aircraft seemed to be no worse off in the 90% equipage case than in the 10% equipage case, which was arguably 
no different than they would be if 0% were equipped and no system benefit was observed. 

4.3 Service for Equipage 

Another decision made for this simulation was to explore the possible application of a "service-for-equipage" 
policy in ANSP decision making at both the multi-sector planning and the sector level, with planners and 
controllers asked to provide priority service to Data Comm equipped aircraft. 

Controller operations. Controllers provided service for equipped aircraft across all three equipage levels by 
making unequipped aircraft yield in mixed equipage conflicts, and by giving equipped aircraft priority access to 
constrained (e.g., by weather) airspace. They indicated in debrief discussions that this was easy to do, although 
they also said that it would often have been easier for them to move an equipped aircraft. 

One reason why equipped aircraft resolutions may have been easier was that Data Comm permitted a clean, one- 
step / one-clearance maneuver to efficiently achieve the weather or conflict avoidance objective. Unequipped 
resolutions, by contrast, often involved a two-step or two-clearance process, regardless of whether the controllers 
used trajectory, heading or altitude clearances. For example, an additional computer entry might be needed to 
bring the ATC system trajectory back into alignment with the flown trajectory; e.g., after the unequipped aircraft 
turns onto its new assigned route. If vectors or an altitude resolution were used, the controller might need to 
remember to return to that aircraft and issue a second clearance later to put it back on course. With trajectory 
clearances, the location of the waypoints sometimes meant that an unequipped aircraft could be sent 
unnecessarily far out of its way, especially when contrasted with the more precise trajectory solutions available for 
equipped aircraft. In order to reduce the impact of these route inefficiencies, controllers would routinely come 
back to the rerouted aircraft several minutes later, after the problem was past, and issue a new clearance to return 
the aircraft back on course. 

Note that these added tasks might contribute to the controller's workload anytime an unequipped aircraft is 
maneuvered. They only present a dilemma in a service-for-equipage context because the controller may forego a 
comparatively easier, equipped aircraft maneuver in order to provide it priority service. 

This raises a second, cognitive, workload issue that was reported by participants: a service-for-equipage policy 
complicates the controller's decision making process. For example, the controller may need to weigh the 
comparative benefit to the equipped flight of moving two unequipped aircraft; or perhaps the unequipped aircraft 
maneuver is less obvious or more complicated. Participants suggested that if the conflict resolution advisory tool 
could automatically provide solutions favoring the equipped aircraft, it would reduce the cognitive overhead of 
working through the available options. 

Concerns about comparative solution efficiency may be less important operationally, however. Controllers in our 
simulation reported that it was easier to move equipped aircraft in mixed equipage, tactical situations, and in 
several mixed equipage conflicts they chose to move the equipped aircraft for this reason. In many (perhaps 
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most?) tactical conflict situations in an operational setting, however, the preference will likely switch, since voice 
enables the controller to confirm that the clearance is understood and will be executed in time. If a voice clearance 
is preferred over Data Comm there is no advantage to moving the equipped aircraft, and there may be a 
disadvantage in taking them off their trajectory and reducing the effectiveness of the TBO tools. 

Planning operations. Area supervisors and traffic management coordinators indicated that they, too, were able 
to provide priority service to the equipped aircraft at all three equipage levels, and they thought it would be 
natural for them to do so in the field as well. They agreed with radar controllers that service for equipage was 
viable on a more strategic level, for solving sector load and weather problems, but perhaps less viable on a very 
tactical level, as an additional constraint for controllers to consider. Traffic management coordinator participants 
also stressed that training for MSP operations needs to emphasize the importance and value of rerouting the 
unequipped aircraft first, otherwise the equipped aircraft would suffer simply because they are easier to move (in 
terms of the number of steps/procedure required to reroute them). With lower equipage levels, it was easier to 
solve traffic problems by moving the unequipped aircraft and leaving the equipped aircraft on their original route. 
As equipage level increased, there were fewer solutions available involving only unequipped aircraft, so some 
equipped aircraft were also moved; however, it's unclear whether that had much impact on their overall 
efficiency, as the equipped aircraft were observed to receive better service at all equipage levels. Moreover, the 
use of lat/long based waypoints allowed equipped aircraft to receive shorter / more precise lateral adjustments to 
their routes at all equipage levels. 

Participant feedback regarding operational suitability. Planners and controllers all found it feasible to 
provide service for equipage, although they felt it was easier, more appropriate, and probably more effective for 
the planners than for the controllers. All agreed that it was an appropriate and useful role for MSP operations, and 
that it could be very effectively performed as part of these operations. In contrast, feedback was divided about 
whether controllers should be expected to provide priority service for equipped aircraft, even though controllers 
reported that it was possible, and that the added workload was not unreasonable. See Appendix G for a 
transcription of relevant material from the post-simulation debrief discussion. Their three main concerns are 
summarized and discussed below. 

One concern was that providing priority service to equipped aircraft - by moving the unequipped aircraft in a 
mixed equipage conflict, for example - would add to controller workload and could compete with other priorities. 
While our participant controllers reported that it would often have been easier to move an equipped aircraft, 
however, they also reported that the workload increase was modest using the tools that were provided. They also 
suggested that resolution advisories would further reduce the effort required. 

A second concern was that asking controllers to provide priority service to a particular aircraft category represents 
a change from their current practice of "first-come, first-served." We observed in the simulation that controllers 
continued to provide service to all aircraft, making an extra effort to minimize reroute inefficiencies for 
unequipped aircraft, for example. The equipped aircraft were favored over the unequipped only when they 
competed for the same resource (e.g., airspace or route access). Priority service was only one of several factors 
considered by controllers when making a control decision, however. On several occasions, for example, a mixed 
equipage conflict was resolved by moving an equipped aircraft nearing its destination instead of an unequipped 
aircraft that was still in mid-cruise. 

Finally, participants questioned whether it was necessary for the controller to actively try to provide benefit at the 
sector level. Our simulation results suggest that the priority-service benefit seen at the sector level may be small 
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compared to the service provided from a multi-sector, or flow management, position. However, further study 
would be needed to understand the differential benefit, and what the impact might be at the sector level. 

4.4 MSP Tools 

Usage data and subjective feedback suggest the prototype tools were effective and satisfactory for mixed 
equipage operations. Both TMCs and controllers developed the majority of their coordinated clearances for 
unequipped aircraft, and the vast majority of these were accepted and executed via voice by controllers. TMCs, 
area supervisors, and controllers gave high usefulness and usability ratings for coordinated clearances and trial 
planning functions for both equipped and unequipped aircraft. 

One of the most critical features for the success of these tools was the integrated trajectory exchange automation 
for development and receipt of coordinated clearances, especially for unequipped aircraft. On the planning end 
there were integrated capabilities that supported trajectory development, evaluation and delivery to controllers, 
via ground-ground data exchange. Integrated functionality continued to be critical on the controller side, with the 
received request integrated with the controller's trial planning automation, once again enabling efficient 
assessment, and then delivery to the aircraft. 

4.5 Procedures 

Several revisions to the MSP concept developed for the 2009 simulation were needed to accommodate mixed 
equipage operations and to explore functional integration of MSP operations into the existing facility workforce. 
Some of the new procedures developed for mixed equipage operations - including methods for providing service 
for equipage, and trajectory-based operations for unequipped aircraft - were discussed in preceding sections. 
Changes to roles and responsibilities within the planning and controller teams are described in this section. 

4.5.1 Roles and Responsibilities 

Multi-sector planning operations. Roles and responsibilities for MSP operations were established through a 
combination of researcher-suggested guidelines and participant input based on domain expertise and preferences, 
and they evolved as participants gained increasing experience in the mixed equipage environment. Although area 
supervisors did use the tools to coordinate a small number of clearances, the vast majority of multi-sector trial 
planning and coordinated clearance requests were performed by the TMCs. Within the TMU, the STMC decided on 
the division of responsibilities between the two TMCs and carried out the necessary within and between center 
verbal coordination. Of the three primary divisions of multi-sector trajectory planning responsibility that were tried 
out over the course of the simulation - by geographic area, task (weather vs. volume), and altitude strata - the 
division by altitude strata was believed by TMCs to lead to the least overlap and duplication of effort, although this 
could depend on equipage level, weather situation, and other factors. 

Controller responsibilities. Radar controllers and associates used the new tools to trial plan and coordinate 
clearances within their own and with other sectors. Radar controllers always maintained responsibility for issuing 
clearances by voice, but clearance delivery by Data Comm was sometimes delegated to the radar associate. Radar 
associates also planned and sent clearances for unequipped aircraft to their own sector controllers to voice up to 
the aircraft. Division of other duties, such as coordinating clearances with other sectors, depended on radar 
controller personal preferences, as well as on the equipage level. At lower equipage levels, radar associates 
assumed more duties because radar controllers were busy voicing clearances, while at the highest equipage level, 
the radar associate might primarily assist with data block management at the higher traffic densities. 
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4.6 MSP Functional Integration: A Comparison to 2009 Simulation 

Integration of MSP tasks within existing workforce. OurtwoTMU participants had both previously 
participated in the 2009 Multi-Sector Planner Simulation, one in each of the two simulation sessions. During that 
simulation, both performed multi-sector planning activities from a dedicated MSP position. Individual planners 
were responsible for non-overlapping geographic areas (e.g., the northeast sectors of ZKC were the purview of 
"MSP North") with a ~30 to 60 minute time horizon, while a single TMC was responsible for the ~45 to 90 minute 
horizon for the entire center. In the current simulation, the TMU assumed responsibility for the 30 to 60 minute 
time window as well, and both TMCs (potentially) shared an overlapping region of responsibility. The STMC was 
responsible for communication and coordination within his own TMU, with outside center TMUs, and with the ZKC 
area supervisors. 

In the post-simulation questionnaire, these two participants were asked to compare the effectiveness of 
operations and team configuration in the earlier simulation with the current study, where they performed multi- 
sector planning functions as a TMC, with an STMC who was largely "off-scope" and available to focus on 
coordination. Both participants agreed that the 2010 team configuration and MSP operations worked well, and 
better than in the 2009 simulation. They also felt they were better able to accomplish their objectives with less 
confusion and duplication of effort. They attributed their improved effectiveness and efficiency to a clearer 
delineation of responsibilities, and oversight of the MSP function by an STMC. STMC management of the MSP 
operations in the TMU, and effective coordination of the delegation of duties were considered key to making the 
operation successful. The fact that the current simulation involved a smaller, "more focused" group was also 
mentioned as an improvement. 

Physical location of planning activities. In the 2009 simulation, the MSPs were physically isolated from the 
rest of the team rather than co-located in the operational area or the TMU. Most participants in 2009 thought the 
MSP would have been more effective from the TMU, however, it was also suggested that an MSP working in the 
operational area might be useful as well. Therefore, the two area supervisor participants in the current study were 
asked whether they could have used a multi-sector planning TMC located in the operational area. Both believed 
the multi-sector planning function and tools would be useful but that a supervisor could handle the 
responsibilities, with no need for a TMC to staff the position. However, one reason to staff the position with a TMC 
might be for ease of coordination with the TMU. Our TMU participants stressed that if MSP functions were to be 
carried out by area supervisors, coordination and communication between area supervisors and the TMU would 
be crucial to avoid duplicating or counteracting each other's efforts. We also observed during the simulation that 
an area supervisor who was "on scope" performing a multi-sector reroute was less effective at monitoring the 
sector controller's task load. However this may have been partially due to their unfamiliarity with the planning 
tools, and with the reduced radio communications in a Data Comm control room which reduces the auditory cues 
to the supervisor. 
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5 Conclusions 


5.1 Tools and Procedures 

An MSP concept tested in 2009 was adapted to support multi-sector planning and planner-sector coordination in a 
mixed equipage environment, then tested in a 2010 follow-up simulation. There were two main changes to 
planning procedures for the 2010 simulation: (1) redistribution of planning functions among STMC, TMCs and area 
supervisors, and (2) differential handling of Data Comm equipped and unequipped aircraft. Sector load assessment 
tools were modified to provide equipage-sensitive complexity feedback that took into account the increased 
controller task load associated with the unequipped aircraft. The main 2010 procedure change, supported by the 
modified assessment tools, was the policy decision for planners to provide the equipped aircraft priority access to 
constrained resources (airspace, routes), selectively moving the unequipped aircraft as needed. This resulted in a 
more workable sector-level problem for controllers, with higher overall throughput and lower workload at the 
higher equipage levels. The new functional distribution of MSP responsibilities also worked well for participants. 
The two who had been part of our earlier simulation said that efficiency and coordination among the planning 
team was greatly improved by consolidating most MSP operations in the TMU. 

Two additional changes that affected controller operations were made for the 2010 mixed equipage simulation. 
The first was the addition of a radar associate to accommodate the increased workload associated with the 
unequipped aircraft. The distribution of responsibilities between radar controller and associate was effective and 
fluid, varying among controller teams and by equipage level. The radar controller always maintained sole 
responsibility for radio voice communication with aircraft, although development of trajectory clearances for 
unequipped aircraft was often delegated to the radar associate. 

It was also necessary to develop a means for coordination of unequipped aircraft trajectory clearances that could 
be easily reviewed and delivered by the controller. Automation tools were modified to enable trajectory clearance 
development and coordination for unequipped aircraft using named waypoint-based trajectories. This automation 
proved effective enough that the function was central not only to MSP operations but also to controller 
operations, for both within-sector and between-sector coordination. It also provided a tool-supported mechanism 
for increasing TBO feasibility for unequipped aircraft. This raises two questions: (1) With suitable tools, how would 
controller workload for issuing trajectory clearances to non-Data Comm aircraft compare to today's vector-based 
operations? (2) If there is a "cost" difference with respect to controller workload, how does this cost compare to 
the benefits of trajectory-based operations? More specifically, what TBO-related benefits are observed by the 
sector controller? With suitable automation, trajectory-based operations may prove a comparable, or even easier 
alternative for controllers than today's methods. If the resulting trajectories are acceptable to users, and if the 
system and controller benefits are significant enough, a modest workload increase may even be acceptable. 

5.2 Service for Equipage 

This simulation provided an opportunity to explore a "best-equipped, best-served" policy of traffic management 
and air traffic control, and the procedures and tools designed to support this objective within MSP and controller 
operations proved both feasible and effective. 

All participants felt that the service for equipage objective was well-suited to MSP operations, and that it could be 
very effectively integrated with flow management or complexity management operations. Solving local area 
demand/capacity imbalances by moving unequipped aircraft away from the affected airspace is a flow planning 
strategy that can maintain higher system throughput while rewarding operators of equipped fleets. Although their 
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initial impulse was to reroute the "easier" equipped aircraft, TMCs were quick to adopt this strategy, and found it 
highly effective, with no apparent downside. 

In contrast, feedback was mixed regarding whether the controller should be asked to provide priority service, even 
though participants reported that it was possible, and that the added workload was not unreasonable. Their 
concerns were: 

• The workload associated with providing priority service may compete with other controller priorities. 

• Priority service is different from current "first-come, first-served" policy. 

• Sector level benefit may be small compared to multi-sector benefit potential, and may therefore be 
unnecessary. 

The concern about controller workload and competing priorities should be satisfied if priority service is only 
expected on a "workload permitting" basis. Our results indicate that if this service is well-supported by 
automation, controllers will likely be able to accommodate it most of the time. 

Implied in the second concern is that the service received by unequipped aircraft would be unsatisfactory, or that 
they would be unfairly penalized. Our simulation observations would suggest, however, that if a policy decision 
were made to provide priority service to equipped aircraft, it could be implemented in a way that limits the impact 
on unequipped aircraft. More important, it could ensure that the system benefits resulting from the introduction 
of Data Comm do not disproportionately accrue to the unequipped aircraft. 

The simulation demonstrates that priority service can be effectively provided at the MSP level, and that MSP 
operations can be designed so that priority access equates to better overall system efficiency. Perhaps a cleaner 
division of responsibilities, where service for equipped aircraft is realized through MSP operations, and that 
controller operations are simply equipage neutral (i.e., do not penalize either equipage type), would be 
satisfactory. Absent further data, however, the answer is unclear. 

In conclusion, it should be emphasized that our results suggest that a service for equipage policy could be feasible 
at both the MSP and sector levels. Feedback suggests that it should be acceptable if implemented and introduced 
in an appropriate way; that is, with good tool support, minimal extra work, demonstrated value, and applied at the 
operator's discretion. 

5.3 MSP Operations with Mixed Equipage 

Overall, results suggested that MSP operations were feasible in a mixed equipage environment and that the tools 
were effective with both equipped and unequipped aircraft. Using the MSP tools, traffic management coordinators 
were able to manage controller task load, effectively balancing throughput with complexity and controller task 
load at each of the three equipage levels tested. Also across equipage levels, mean reported task and workload 
remained tolerable, and operational acceptability was reported to be satisfactory. Although reported frustration 
and confusion were comparatively higher at the 50% equipage level than at the other levels, overall the 50% mix 
was believed to be workable. 

Benefits were observed both in terms of system performance and operational support for a "best-equipped best- 
served" approach to traffic management and air traffic control. As equipage level increased, throughput increased, 
even as controller workload decreased. MSP operations effectively supported priority service for equipped aircraft; 
more equipped than unequipped aircraft were routed through the test airspace and unequipped aircraft received 
a greater increase in flight path length. 


MSP3_Report_3-2-ll 


3/2/2011 


58 


Other operational procedures established throughout the simulation suggested that the bulk of multi-sector 
planning - that is, trial planning and clearance coordination - can be effectively carried out by the traffic 
management unit (TMU). Operational area supervisors may also perform these functions, though far less 
frequently. Within the TMU, the division of specific MSP roles and responsibilities by the supervisory traffic 
management coordinator (STMC) among the traffic management coordinators (TMCs) remained flexible, with 
divisions by altitude strata, geographic area, and airspace problem (e.g, weather constraint or traffic volume) all 
possible contingent on the situation. On a more tactical scale, radar controllers and associates also found the 
clearance coordination tools useful and effective for trial planning and coordinating clearances within their own 
sector team and with other sectors. Other than voice communication with aircraft, which was always performed by 
the radar controller, the division of roles and responsibilities between radar controllers and associates varied by 
sector team and by equipage level. 

In summary, the MSP concept, as prototyped and tested in this simulation, appeared to be both feasible and 
beneficial in a mixed equipage environment when used to manage sector complexity and convective weather 
reroutes. A framework of coordination procedures supported by integrated decision support and communication 
tools supported collaborative trajectory management, with strategic clearances developed in response to varying 
constraints from a non-controller station. This framework could be extended to support other flow management 
functions, such as point-in-space metering and arrival flow management, and provides a model for how to 
integrate strategic trajectory management with air traffic control operations. 
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Acronyms 


ANOVA 

ANSP 

AOL 

ATC 

BEBS 

CARS 

CC 

CID 

CPA 

D, D-Side 

Data Comm 

DSR 

FAA 

FDB 

FL290 

FMS 

HITL 

HOST 

LOS 

MAP 

MSP 

NAS 

NASA 

NASA TLX 

NextGen 

NRS 

OE 

PE 

QC 

R, R-Side 
SA 

STMC 

Sup 

TBO 

TMC 

TMU 

TR 

TSD 

Tukey's HSD 

UC 

VSCS 

WAK 

Wx 


Analysis of Variance 

Air Navigation Service Provider 

Airspace Operations Laboratory at NASA Ames Research Center 

Air Traffic Control 

Best Equipped Best Served Policy 

Controller Acceptance Rating Scale 

Coordinated Clearance 

Computer identification number, 4 digit aircraft identifier used for command entries 

Closest Point of Approach 

Radar Associate Controller 

Air-Ground Data Communications 

Display System Replacement (radar display) 

Federal Aviation Administration 
Flight Data Block 
Flight Level 29,000 Feet 
Flight Management System 
Human-ln-The-Loop Simulation 

En Route Center Computerized Flight and Data Processing System 

Loss of Separation 

Monitor Alert Parameter 

Multi-Sector Planning 

United States National Airspace 

National Aeronautics and Space Administration 

NASA Task Load Index 

Next Generation Air Transportation System 

Navigation Reference System 

Operational Error 

Proximity Event 

ATC System Data Amendment 

Radar Controller 

Situation Assessment 

Traffic Management Coordinator Supervisor 
Supervisor 

Trajectory-Based Operations 

Traffic Management Coordinator 

Traffic Management Unit 

Trial Plan input with named waypoint(s) 

Traffic Situation Display 
Honestly Significant Difference Test 
Uplink Clearance 

Voice-Switching and Communication System 

Workload Assessment Keypad 

Weather 
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Appendix A: MSP and Controller Input Commands 


Two-character input commands for controller workstations. 


2-Char 

Command 

Syntax [Type in CRD] followed 
by ENTER 

Function 

Trial Planning Functions 

TT 

TT [CID] 

Open trial plan for [CID] 


TT 

Turn off all trial plans 

TA 

TA [altitude] [CID] 

Trial plan an altitude for [CID] (in flight levels) 

TR 

TR [fix] [CID] 

Open trial plan route direct to [fix] for [CID] (trial plan route then rejoins 
original routing) 

Data CommFunctions | 

UC 

UC [CID] (with trial plan or CC) 

Uplink trial plan or CC request to [CID], Updates ATC computer system. 

cc 

CC [CID or SEL] 

Coordinates trial plan for [CID or SEL] to ATC sector with track control of [CID] 

CN 

CN [CID] 

Sends non-positive response to sender regarding proposed trial plan of [CID] 

CY 

CY [CID] 

Sends affirmative response to sender regarding proposed trial plan of [CID] 

QC 

QC [CID] 

Assign trial plan as a ATC computer system amendment for unequipped [CID], 

DE 

DE [CID] (or) DE /OK [CID] 

Delete all closed Data Comm messages for [CID] from status list 

| Other Sector Quick Actions | 

ID 

ID [CID] 

Highlights [CID] 

QL 

QL [sector #] 

Quick look for [sector #] 

QP 

QP [sector #] [CID] 

Point-Out [CID] to [sector #] 

QPJ 

QPJ [CID] 

Toggle display of j-ring for [CID] 

QF 

QF [CID] 

Display flight plan readout for [CID] 

QU 

QU [CID] 

Display FMS route for [CID] 

FX 

FX [3-letter ID or fix name] 

Displays Fix on DSR and shows name and 3-letter identifier in CRD 


Two character input command commands for planner (supervisor or TMC) workstations. 


2-Char 

Command 

Syntax [Type in CRD] followed 
by ENTER 

Function 

Trial Planning Functions 

TT 

TT [CID] 

Open trial plan for [CID] 


TT 

Turn off all trial plans 

TA 

TA [altitude] [CID] 

Trial plan an altitude for [CID] (in flight levels) 

TR 

TR [fix] [CID] 

Open trial plan route direct to [fix] for [CID] (trial plan route then rejoins 
original routing) 

FF 

FF [ACID or CID] [CID] [CID] etc 
FF [SEL] or FF [000] 

Selects multiple data tags for altitude or route plan changes 


FF 

Turns off all multi-aircraft selections 

| Data Comm Functions | 

UC 

UC [CID] (with trial plan shown 
or CC message) 

Uplink trial plan to [CID] and trial plan will be entered into ATC system. 

CC 

CC [CID or SEL] 

Coordinates trial plan for [CID or SEL] to ATC sector with track control of [CID] 

CP 

CP [MSP or TMU position#] 
[CID or SEL] 

Coordinates trial plan for [MSP or TMU position #] of [CID or SEL] 

CN 

CN [CID] 

Sends non-positive response to sender regarding proposed trial plan of [CID] 

CY 

CY [CID] 

Sends affirmative response to senderregarding proposed trial plan of [CID] 

QC 

QC [CID] 

Assign trial plan as a ATC system amendment for unequipped [CID]. 


Additional commands for planner workstations that filter traffic presentation. 
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Filter 

Commands 

Syntax [Type in CRD] followed by 
ENTER 

Function 

FC 

FC [TO, FROM, VIA, FL, GEO, 
DRAW, WX, CON, ACID, AIRLINE, 
FR, LOAD] 

Adds filter commands to the aircraft filter list on the DSR 

TO 

FC TO [airport] or [ARTCC] 

Filter aircraft to specific arrival airport(s) 

FROM 

FC FROM [airport] or [ARTCC] 

Filter aircraft coming from a specific airport(s) 

VIA 

FC VIA [fix] 

Filter aircraft going via a certain waypoint/fix 

FL (ALT.) 

FC FL [alt] [alt] 

Filter by altitude(s) 

GEO (SECTOR) 

FC GEO [ZKC90] or [ZME] [T] 

Filter by sector ownership or ARTCC @ Time X 

DRAW or LINE 

FC DRAW or LINE [FI] [T15-35] 

Filter aircraft that will enter any "Draw Tool" defined area @ Time X 

WX 1, 2, 3 

FC WX 1,2,3 [T25-45] 

Filter aircraft that are predicted to go into weather low (1), medium (2), 
and high (3) @ Time X 

CONFLICT 

FC CON Tl=30 

Filter aircraft that are predicted to be in conflict at Time X (Tl-30, 
between now and 30 minutes) 

ACID 

FC ID [NWA123] 

Filter by ID (NWA123) 

AIRLINE 

FC AIRLINE [SWA] 

Filter by airline (SWA) 

AIRPORT 

FC AIRPORT [DFW] 

Filter aircraft to/from this airport 

DIR 

FC DIR [Heading Range 045-090] 

Filters aircraft heading in a specific direction 

LOAD 

FC LOAD 

Filter based on Load Table/Graph criteria selected 

FR 

FC FR [IFR or TFR] 

Filter based on equipage: Equipped (TFR) and Unequipped (IFR) 
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Appendix B: Participant Briefing Material 


Slide content from participants' May 18, 2010 "Morning Briefing: Introduction" 


Multi-Sector Planning for Mixed Equipage Airspace 

> What do we mean by Multi-Sector Planning (MSP)? 

• Process for solving local area problems by modifying trajectory of one or more 
aircraft 

• Performed in the TMU and/or on the control floor 

> Nominal roles and responsibilities: 

• Area supervisor and traffic management monitor local traffic situation 

• TMC plans trajectory changes, coordinating with supervisor and others 

• Supervisor manages plan execution by controllers 

• Controllers review trajectory requests and execute if suitable 

> MSP and mixed equipage: 

• Trajectory-based solutions developed for unequipped and equipped aircraft. 

• Unequipped aircraft are constrained to waypoint-based trajectories; equipped 
aircraft are not. 

• MSP supports NextGen "Best-Equipped Best-Served" objective: Equipped aircraft get 
preferred access to constrained airspace. 


Slide content from participants' May 18, 2010 "Afternoon Briefing: Roles and Responsibilities" 


"Best-Equipped, Best-Served" Priority Service Means: 

> For Controllers /Area Supervisor: 

• Conflicts: 

o If equipped is in conflict with unequipped, move the unequipped when able, 

o Negotiate resolutions with adjacent sectors as needed. 

• Sector Load or Complexity: When able, let equipped "fly through” and reroute 
unequipped. 

• Weather: Let equipped fly when able; only move to skirt around the weather. 

> For TMU: 

• Complexity: If sector complexity is high, re-route unequipped when able. 

• Sector Load: Let equipped "fly through" the constrained area, reroute unequipped out 
of loaded sector when able. 

• Weather: 

o Equipped: When able, let equipped fly, only move to skirt around the weather, 

o Unequipped: Move to give wide berth to weather cells using fix/waypoint-based routes. 
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Roles and Responsibilities for TMU Team 

> STMC: Manage facility TMU operations, including: 

• Coordinate with Command Center, neighboring TMUs, and Area Supervisors 

• Monitor capacity/demand balance in area of responsibility. 

• Assign TMC position(s) as needed to manage local multi-sector trajectory changes, (e.g. weather 
reroutes, load capacity issues etc...) 

• Coordinate TMC activities and communicate as needed. 

> TMCs: Perform tasks/roles as designated by STMC, e.g.: 

• Monitor and manage complexity in assigned area of responsibility. 

• Develop local area flow modifications in response to external events or situations. 

• Provide priority service to equipped aircraft when modifying trajectories or flows. 

• Coordinate as appropriate on management of developing situation... 

• Clearances may be sent directly to A/C if first route change is more than 30 minutes out. 


Roles and Responsibilities for Area Supervisor 

> Use available automation to monitor sector load and complexity within area, and inform TMU and 
affected sectors of situations as appropriate. 

> Coordinate with the TMU to develop local area plans. Use "best-equipped best-served” protocol 
when reroutes are necessary. 

> Coordinate with adjacent Area Supervisors for local load and complexity. 

> Inform controllers about TMU reroutes. 

> Coordinate as needed regarding plan execution. 

> Monitor plan execution, and inform TMU when plan cannot be executed. 

> Assign D-Side controllers as needed. 


Roles and Responsibilities for Sector Team 

> Team Responsibilities: 

• Coordinate regarding D/R division of responsibilities (e.g., with respect to clearance requests, 
development of clearances for equipped or unequipped aircraft, handoffs, data block 
management). 

• Best-equipped best-served: Provide priority service to equipped aircraft when able. 

• Coordinated Clearance (CC) Requests: 

o Review external CC requests (from TMC or Sup), and execute when able, 
o Modify if needed to resolve traffic conflicts, 

o Inform supervisor if CC request is rejected. 

> R-Side controller: 

• Team leader. 

• Maintains radio communication with all aircraft. 

> D-Side controller: 

• Coordinate as needed, including point-outs. 
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Nominal D-Side Tasks 

> D-Side acts as a second set of eyes 

• Look for conflicts, especially further out to see conflicts that involve multiple sectors 

• Look to ensure that handoff initiation/acceptance are done in time. 

> D-Side performs coordination tasks 

• Coordinate conflict resolutions with adjacent sector as needed 

• Get aircraft that are not on frequency 

> D-Side executes whatever R-Side requests 

• Execute tasks that distract R-Side from their primary responsibilities (e.g., route modification, 
handoffs, data block management, etc.) 


Possible Additional D-Side Tasks 

D and R can divide these tasks globally or on a plane-by-plane basis, at radar controller's discretion. 

> For equipped aircraft: 

• Reroutes around weather: 

o Reroute around weather and send clearance to the flight deck, or 
o Construct wx avoidance route and CC to R-Side 

• Resolve conflicts: 

o Resolve conflict and send clearance to the flight deck, or 
o Construct conflict-free route and CC to R-Side (or ghost) 

> For unequipped aircraft: 

• Reroutes around weather: 

o Reroute around weather, then 

• send CC (ground-ground data coordination) to R-Side for review, or 
■ verbally coordinate the constructed route, then amend the ATC system's flight plan 
o R-Side issues the route clearance by voice. 
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Appendix C: Modified NASA-Task Load Index (NASA-TLX) 


Immediately following each run, radar controllers and associates, area supervisors, and TMU participants 
responded to the following six questions. 

1. How much mental activity was there for you in the last run? (e.g., thinking, deciding, calculating, remembering, 


looking, searching, etc.) 

Please click on a number to choose your rating. 
1 2 3 

4 

5 

6 

7 

Very low 
mental activity 

Average 



Very high 
mental activity 


2. How much physical activity was there for you in the last run? (e.g., how much did you use your keyboard, mouse, 
and radio for voice communications, if applicable, etc.)? 

Please click on a number to choose your rating. 

1 2 3 4 5 6 7 

Very low activity Average Very high activity 

3. How successful do you think you were in accomplishing the goals of the task? 

Please click on a number to choose your rating. 

1 2 3 4 5 6 7 

Very low success Average Very high success 

4. How hard did you have to work mentally and physically to accomplish this level of success? 

Please click on a number to choose your rating. 

1 2 3 4 5 6 7 

Very little effort Average A lot of hard work 

5. How much time pressure were you under? (Did you feel rushed and that you did not have enough time to 


complete tasks? Or that you did not have enough to do?) 
Please click on a number to choose your rating. 

1 2 3 4 

5 

6 

7 

No time pressure Average 

at all 



Considerable time 
pressure 


6. Were you frustrated by this run? (e.g., were you discouraged, irritated, stressed, and annoyed, or were you 
content, relaxed, gratified, and complacent when performing the task?) 

Please click on a number to choose your rating. 

1 2 3 4 5 6 7 

Low frustration Average High frustration 
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Appendix D: Modified CARS 



Radar Controller Acceptability Rating Scale 


Improvement mandatory. 


r a 

Adequate performance is 
not achievable with 
tolerable workload 
levels. Deficiencies are 
unreasonable. 

v y 


Improvement is needed. 
Deficiencies warrant 
further improvement. 


Improvement mandatory. The traffic could 
not be managed using the procedures. 



Major deficiencies. Control of current traffic was 
only possible by neglecting some traffic 
management tasks and situation awareness was 
severely degraded. Effective teamwork and 
communication could not be achieved. 

CM 

Major deficiencies. Control of current traffic was 
not compromised but maintenance of situation 
awareness in the position was marginal. 
Performance was marginal, as considerable 
work-arounds were required to achieve minimal 
traffic management. Marginal levels of teamwork 
and communication were possible but with 
significant increases in workload. 

CO 

Major deficiencies. Control and situation 
awareness of traffic could be maintained but 
adequate performance was achievable only with 
significant effort. Adequate levels of teamwork 
and communication could only be achieved with 
significant increases in workload. 

4 


Significant deficiencies. Adequate performance 
required hard work and considerable work- 
arounds. Adequate levels of teamwork and 
communication were achieved. 

5 

Moderate deficiencies. Procedures required 
considerable work-arounds to adequately 
manage the traffic. 

CD 

Minor, but annoying, deficiencies. Desired traffic 
management required moderate work-arounds. 

7 


Mildly unpleasant deficiencies. Position was 
acceptable and minimal work-arounds were 
needed to meet desired performance 
effectiveness. 

00 

Negligible deficiencies. Position was acceptable 
and work-arounds were not a factor to achieve 

Q 

desired performance effectiveness. 


Deficiencies were few. Position was acceptable 

10 

and no work-arounds were needed to achieve 

desired performance effectiveness. 


How well does this 
description and 
rating (out of 10) 
match what you 
think was the 
acceptability of the 
operations? 

Please circle one: 

Well 

OK 

Not well 

If you circled “not 
well,” please 
explain why the 
rating feels wrong. 
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start Area Supervisor Acceptability Rating Scale 



How well does this 
description and 
rating (out of 10) 
match what you 
think was the 
acceptability of 
these operations? 
Please circle one: 

Well 

OK 

Not well 

If you circled “not 
well,” please 
explain why the 
rating feels wrong. 
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start (S)TMC Acceptability Rating Scale 



How well does this 
description and 
rating (out of 10) 
match what you 
think was the 
acceptability of 
these operations? 
Please circle one: 

Well 

OK 

Not well 

If you circled “not 
well,” please 
explain why the 
rating feels wrong. 
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Appendix E: Radar Controller and Radar Associate 
Tools Questionnaire Data 


Tables include responses from participants and confederates who staffed the four ZKC test sectors. 

Ratings 1 to 6: Not at all Useful/Usable (1) to Very Useful/Usable (6) 


Part 1. Radar Controller and Radar 
Associate Display Synchronization 

Radar Controllers (n=4) 

Radar Associates (n=4) 


Useful 

Average 

Usable 

Average 

Useful 

Average 

Usable 

Average 

Graphical Flight Plan Readout 

5.00 

5.50 

4.00 

4.50 

Halo/J-ring 

6.00 

6.00 

5.75 

6.00 

Flight Data block position 

5.50 

5.75 

5.75 

5.25 

Datalink UC (up arrows next to callsign) 

6.00 

6.00 

5.75 

5.50 

Datalink UC (message in status list) 

5.25 

3.75 

3.75 

3.00 
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Part 2. Mixed Equipage Controller Tools 
(continued) 

Radar Controllers (n=4) 

Radar Associates (n=4) 

Equipped 

Unequipped 

Equipped 

Unequipped 

Mixed Equipage Communication Tools 

Useful 

Average 

Usable 

Average 

Useful 

Average 

Usable 

Average 

Useful 

Average 

Usable 

Average 

Useful 

Average 

Usable 

Average 

Transfer of Communication 
(E-Auto; UE-Manual) 

6.00 

6.00 

4.00 

3.33 

6.00 

6.00 

5.50 

4.50 

Mechanism for Building Trial Plans (E-Lat/Long; 
UE-Snap-to Fix) 

6.00 

6.00 

4.75 

3.75 

6.00 

6.00 

5.00 

4.00 

Transfer of Clearance 

(E-UC Uplink clearance; UE- QC ATC system 
amemdment) 

6.00 

6.00 

6.00 

5.25 

6.00 

5.75 

5.50 

4.75 

Mechanism for Receiving/Detecting Clearance 
Requests from TMC 

5.75 

5.75 

5.75 

5.75 

5.75 

5.25 

5.75 

5.25 

Mechanism for Reviewing Clearance Requests 
from TMC 

5.75 

5.25 

5.75 

5.25 

6.00 

6.00 

6.00 

5.50 

Mechanism for Receiving/ Detecting Clearance 
Requests from a RA/R-Side 

5.75 

5.75 

5.75 

5.75 

6.00 

5.75 

5.75 

5.75 

Mechanism for Reviewing Clearance Requests 
from a RA/R-Side 

5.75 

5.75 

5.75 

5.75 

6.00 

6.00 

6.00 

5.75 

Communication Drop-down Window 

4.75 

5.50 

4.50 

5.50 

5.25 

5.25 

5.25 

5.25 

CY (Clearance Yes - WILCO) 

4.00 

4.00 

4.00 

3.67 

6.00 

6.00 

4.50 

4.50 

CN (Clearance No - UNA) 

4.00 

4.00 

4.67 

3.67 

6.00 

6.00 

4.50 

4.50 
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Appendix F: MSP3 Tools Questionnaire Data for STMC/TMC and SUP 


Ratings 1 to 6: Not at all Useful/Usable (1) to Very Useful/Usable (6) 


Part 1: Traffic Monitoring and Problem 
Identification 

STMC/TMC 

(n=2) 

SUP 

(n=2) 

Load Display Control Window 

Useful 

Average 

Usable 

Average 

Useful 

Average 

Usable 

Average 

Cell Values 

TOTAL 

5.00 

3.50 

3.00 

6.00 

PEAK 

5.50 

5.50 

6.00 

6.00 

AVERAGE 

3.00 

3.00 

2.50 

6.00 

PEAK/ 

TOTAL 

4.00 

3.50 

3.00 

6.00 

Categories 

ALL 

5.50 

5.00 

3.00 

6.00 

CNFLT_ 

CNT 

4.00 

5.00 

1.50 

6.00 

CNFLT_AC 

4.00 

5.00 

1.50 

6.00 

TRANS 

3.00 

3.00 

2.50 

6.00 

FILTR 

3.50 

3.50 

2.50 

6.00 

UNEQP 

4.25 

5.00 

N/A 

6.00 

WETHR 

4.50 

4.50 

2.50 

6.00 

CMPLX 

5.00 

5.50 

6.00 

6.00 

Show Category only 

5.00 

4.00 

2.50 

6.00 

Show Category & ALL 

5.00 

5.00 

5.00 

6.00 

Selection Logic 

Single Cell 

4.00 

4.00 

6.00 

6.00 

Multi Cell 

6.00 

6.00 

5.00 

6.00 

In any cell 

5.50 

5.50 

4.00 

6.00 

In all cells 

4.00 

5.00 

4.00 

6.00 

Overall Usefulness and Usability of Load Table Window 

6.00 

5.50 

6.00 

5.50 

Color coded cell values (red, yellow, green) 

6.00 

6.00 

6.00 

6.00 

Color coded trial plan values(cyan numbers) 

6.00 

6.00 

5.00 

6.00 

Sector Boundary 

Highlighting ("*" left of sector ID) 

4.00 

5.50 

4.00 

6.00 

Cell Selection (Magenta box) 

6.00 

6.00 

6.00 

6.00 

Single Sector Graph Selection (by clicking on "G" left of sector id) 

3.50 

4.00 

5.00 

6.00 
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Part 1. Traffic Monitoring and Problem Identification (continued) 


STMC/TMC 

SUP 

(n=2) 

(n=2) 


cu 

<U 

CU 


CU 

cu 

CU 
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dfl 
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dfl 
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_D 
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<D 

t/l 
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cu 

CD 

co 

cu 
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D 
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< 

D 

> 
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D 
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Overall Usefulness and Usability of Load Graphs 

5.50 

5.50 

6.00 

6.00 

Color Coding 

5.50 

5.50 

5.50 

6.00 

Selectable Time Slice 

5.50 

5.50 

6.00 

6.00 


Overall Usefulness and Usability of Traffic Situation Display (TSD) 

3.00 

3.50 

5.00 

5.00 

Weather Depiction (color coding) 

4.50 

4.50 

5.00 

6.00 

Weather Loop: History 

4.50 

4.50 

5.00 

6.00 

Weather Loop: Forecast 

4.50 

4.50 

6.00 

6.00 

Traffic Depiction 

4.50 

4.50 

4.50 

6.00 


Overall Usefulness and Usability of AC Filters 

5.00 

4.00 

6.00 

5.50 

Draw/Line Filter 

3.50 

3.50 

3.00 

6.00 

VIA Filter 

3.50 

3.50 

3.00 

6.00 

ACID Filter 

3.50 

3.50 

3.00 

6.00 

FL Filter 

4.50 

5.00 

3.00 

6.00 

Conflict Filter 

3.00 

3.50 

5.00 

6.00 

TO/FROM Filter 

4.50 

3.50 

4.00 

6.00 

Load Filter (using cells in the load table) 

5.00 

5.50 

6.00 

6.00 

Airline Filter 

3.50 

3.50 

4.00 

6.00 

GEO Filter 

4.50 

3.50 

4.00 

6.00 

WX Filter 

4.50 

3.50 

6.00 

6.00 

Load Filter (using time slice in the load graph) 

4.50 

3.50 

6.00 

6.00 

Color Coding (of selected filter setting) 

4.00 

2.00 

5.00 

6.00 

Equipage Filter (unequipped or equipped) 

5.00 

5.50 

3.00 

3.00 
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Part 2: Solution Planning 

STMC/TMC 

( n =2) 

SUP 

( n =2) 

Tool Set 

Useful 

Average 

Usable 

Average 

Useful 

Average 

Useable 

Average 

Traffic Situation Display 

3.50 

2.50 

4.50 

6.00 

DSR Traffic View Display 

5.50 

5.00 

5.00 

6.00 

Load Table 

5.50 

5.00 

6.00 

6.00 

Load Graphs 

5.00 

4.50 

6.00 

6.00 

AC Filters 

5.50 

5.00 

5.00 

6.00 

Trial Planning 





FF (Group selection for group planning) 

5.50 

5.00 

5.00 

6.00 

TT (Trial Planning) 

4.00 

4.50 

6.00 

6.00 

TA (Altitude Trial Planning) 

5.00 

5.00 

5.00 

6.00 

TR (Route Trial Planning) 

5.50 

5.50 

5.50 

6.00 

Conflict Probing in real-time 

4.50 

4.00 

4.50 

6.00 

Weather penetration probe in real-time 

5.50 

4.00 

5.50 

6.00 

Graphical Trial Planning for Equipped aircraft 

6.00 

6.00 

3.00 

6.00 

Graphical Trial Planning Unequipped aircraft 

6.00 

6.00 

3.00 

6.00 

Communication Set 

Useful 

Average 

Usable 

Average 

Useful 

Average 

Usable 

Average 

Voice Communication 





1-to-l Voice Communication 

5.50 

5.50 

5.00 

6.00 

1-to-Many voice communication 

5.50 

5.50 

5.00 

6.00 

Data Comm 





UC (Uplink clearance) 

5.50 

5.50 

6.00 

6.00 

CC (send coordinated clearance to radar controller) 

5.50 

5.50 

6.00 

6.00 

CP (Copy coordinated clearance to others) 

5.50 

5.50 

6.00 

6.00 

CY (Clearance Yes- WILCO) 

4.00 

3.00 

4.00 

6.00 

CN (Clearance No- UNABLE) 

6.00 

6.00 

4.00 

6.00 
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Appendix G: Excerpt from Post-Simulation Debrief Discussion 


The material below was transcribed from the post-simulation debrief discussion, and edited slightly for clarity. The 
selections includes participant responses to questions about providing priority service for the Data Comm- 
equipped aircraft. 



SECTION 1: Discussion with Controller Participants and Supervisor 

Question (NS): 

"\Ne asked you to always try to provide better service to the data comm equipped aircraft. 1 was 
wondering what you thought of that: Was that was an unreasonable request ? Were you able to do it for 
the most part? How did it go?" 

Controller 1: 

"[In most mixed equipage conflicts] it would've been much easier to turn the equipped aircraft, but since 
you asked us to try to leave them alone, we moved the unequipped, and it wasn't all that difficult. It was 
a little more awkward, but nothing 1 couldn't handle. If 1 were a controller on the floor, in an operational 
setting, 1 guess I'd want to understand why, because, if the operation is [trajectory based], as it was up in 
the simulation, it's easier to move the equipped aircraft from point to point, instead of moving the 
unequipped from fix to fix. " 

Question (NS): 

"So how much do you think that added to your workload?" 

Controller 1: 

"It did add some, because sometimes it was awkward finding a suitable fix. For example, where you have 
to go way around weather and you're searching for something that will keep him close to the weather, 
and it's way downstream. Or sometimes you have to pick a fix that's way past the weather. Then you 
have to return to that aircraft later because you don't want to send him too far out of his way. Once he's 
past the weather you can clear him short back to the next fix, but that means you need to get back to him 
in three, four, or five minutes later, whenever, when you think about it." 

Controller 2: 

"1 did it slightly differently. In most cases it didn't matter which one 1 moved, but there were a couple 
where there wasn't a good fix.. .so 1 ended up vectoring them. 1 wouldn't change the route, 1 just vectored 
him until he was clear of the traffic, then 1 put him right back on course, with a TR [route amendment] 
back to that next fix. Or 1 would vector him around the weather and then TR him back to the fix.. .That was 
a workaround, when there wasn't a suitable NAVAID, or 1 couldn't find one without having to range way 
out..." 

Question (NS): 

"And that worked out ok?" 

Controller 2: 

"For me it did. It added a little bit of workload because you had to remember, I've got this guy on a 
vector, 1 don't want to all of a sudden look down and there he is..." 

Question (NS): 

"Supervisors and traffic managers, what did you think about trying to prioritize the equipped aircraft?" 

Supervisor 1: 

"From a Sup's standpoint, it wasn't any big deal. Because when 1 was selecting anybody to do anything to 
[1 could pick] the unequipped first, ... [because] there was no time crunch for me. You know 1 wasn't 
feeling the pressure of talking to twenty-five airplanes. [1 had more time to explore what would work], ... 
it wasn't really that big of a deal in my position." 

SECTION 2: Later Discussion with Traffic Management Participants and others 

Confederate TMC: 

"1 would say though, the weather and the loading, definitely. Best-equipped, best-served. The biggest 
push back on BEBS will be from a controller, "You're telling me who 1 have to turn." You're gonna get 
some severe resistance on that. But as far as the sector loading and the weather reroute, no problem." 

S/TMC 1: 

"1 agree. He's absolutely right about that. 1 mean you tell a controller you have to move this guy instead 
of this guy; there's your push back. BEBS has to occur outside of the controller position (at the strategic 
level, MSP/TMC position). Once you get within that immediate time-frame, the controller's gonna have 
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Question (TP): 


S/TMC 1 
Question (AK) 
S/TMC 1 
Question (AK) 
Confederate TMC 
? 

? 

Question (TP) 
Confederate TMC 
Speaker A 


Speaker B: 
Speaker A: 

Question (TP): 

Speaker B: 

Speaker A: 

so..." 

Speaker B: 
Speaker A: 


to make the decision. ... [MSP/TMC can do BEBS and the controller can uplink the CCs, but the BEBS 
should be transparent to the sector controller.] The [situation] changes when an a/c is coming up on 
weather or on another a/c and you have to make a decision about who has to be turned, and you're 
supposed to be thinking Best Equipped Best Served, that's not going to happen. The controller has to 
make that decision based on what's more efficient right then right now, and what's safe." 


"...if the policy came out, ok so now we're introducing this, there's gonna be equipped aircraft in there 
and all the controllers were told, "Please give them preferential service unless you need to move them for 
... unless you're compromising safety," do you think that they really wouldn't buy into this? What would 
you expect would happen ?" 

"It happens now. Wait we're doing that now." 

"BEBS happens now?" 

"Heck yeah!" 

"What?" 

"Yeah, they wouldn't buy into it." 

"Maybe after 3 years (?)" 

"Yeah, they would buy into it eventually. " 

"They would not you said?" 

"There would be push back. " 

"But if we back up a little bit, again the MSP tools, and this is the first time I've been exposed to it, it looks 
to me the best bang for the buck is 30 to 45 minutes ahead of the impacted sector. So if TMU or MSP 
position is already moving the unequipped aircraft, what the controller's doing in the sector, I'm not sure 
those little 10 degree turns make that much difference. Maybe we’re already supplying Best Equipped 
Best Served. Because we’re already doing it before they get into the sector. " 

"Right. That's the key." 

"We're doing it Traffic Flow Management-wise. Again a controller has to move a plane based on the 
position of the plane. And they're not gonna look and say, well, wait, that one's not equipped. " 

"But you did it here. " You looked when there was a mixed conflict, you tried to move the unequipped, 
because we asked you to, so..." 

"But it's different here, in the simulation. Since you have a bunch of supervisors working the scenario for 
research. " 

"It also depends on whether they'll have SA and the conflict probe available to them. " 


"But it's different here, in the simulation. Since you have a bunch of supervisors working the scenario for 
research. " 

"It also depends on whether they'll have 5A and the conflict probe available to them." 
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